Leaderboard

Official SemEval Results on the Test Set

English - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	DUTIR	0.41280	0.48100	0.43020	0.45420	0.94890
2	PATeam	0.38300	0.46150	0.43020	0.44530	0.88940
3	DEMON	0.37450	0.44870	0.39620	0.42080	0.91910
4	gowithnlp	0.37020	0.44260	0.39250	0.41600	0.93620
5	TartanTritons	0.35740	0.11440	0.10190	0.10780	0.72340
6	Fane	0.34470	0.40850	0.36230	0.38400	0.89360
7	QUST	0.32770	0.39040	0.36980	0.37980	0.92340
8	LATeIIMAS	0.31060	0.36710	0.32830	0.34660	0.83830
8	NlpUned	0.31060	0.37870	0.33580	0.35600	0.84680
9	m1nadzuki	0.25960	0.31490	0.27920	0.29600	0.87660
10	LTG	0.25530	0.07490	0.07550	0.07520	0.58300
11	BERTastic	0.25110	0.31490	0.27920	0.29600	0.86380
11	adithjrajeev	0.25110	0.32140	0.30570	0.31330	0.86810
12	Mekky	0.21700	0.29480	0.27920	0.28680	0.80000
13	NarrativeMiners	0.21280	0.26810	0.23770	0.25200	0.68940
14	FromProblemImportSolve	0.20430	0.29030	0.30570	0.29780	0.77450
15	YNUzwt	0.20000	0.26380	0.23400	0.24800	0.52340
16	Cimba	0.18720	0.23950	0.21510	0.22660	0.09790
17	NarrativeNexus	0.18300	0.20850	0.18490	0.19600	0.71910
18	Rosetta	0.17870	0.22920	0.21890	0.22390	0.60430
19	Dhananjaya	0.17450	0.22880	0.20380	0.21560	0.81280
20	UMZNLP	0.14040	0.21430	0.23770	0.22540	0.70640
21	Tuebingen	0.13190	0.20580	0.21510	0.21030	0.74040
22	YNUHPCC	0.08940	0.11490	0.10190	0.10800	0.75320
23	north	0.08510	0.25870	0.42260	0.32090	0.94890
24	HowardUniversityAI4PC	0.08090	0.08320	0.17740	0.11330	0.69360
25	kzeky	0.06810	0.08090	0.07170	0.07600	0.56600
26	bumblebeeTransformer	0.06380	0.13640	0.16980	0.15130	0.78300
26	eevvgg	0.06380	0.10550	0.17360	0.13120	0.78720
27	Baseline	0.03830	0.04680	0.04150	0.04400	0.28510
28	Team12	0.02130	0.07670	0.09810	0.08610	0.72340
29	cocoa	0.01700	0.08750	0.12450	0.10280	0.82550
30	SemanticaInnovators	0.01280	0.04680	0.16600	0.07300	0.25110
31	TechDacians	0.00000	1.00000	0.00000	0.00000	0.09790
32	teamninebonn	0.00000	0.03260	0.08680	0.04740	0.17870

English - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	GATENLP	0.59000	0.35300	0.43800	0.33300
2	COGNAC	0.55400	0.40000	0.42600	0.39100
3	INSALyon2	0.51300	0.37800	0.40600	0.38200
4	23	0.49300	0.39200	0.37700	0.38400
5	NCLteam	0.48600	0.36300	0.34500	0.36000
6	Narrlangen	0.44400	0.41100	0.34400	0.38600
7	PATeam	0.52100	0.35600	0.33900	0.29100
8	YNUzwt	0.48500	0.37800	0.32100	0.33200
9	iLostTheCode	0.46700	0.35600	0.32000	0.32100
10	Narrengers	0.45700	0.36300	0.31800	0.31800
11	UNEDTeam	0.51200	0.36400	0.31300	0.29400
12	CtrlAltElite	0.47400	0.42200	0.31100	0.44600
13	NotMyNarrative	0.40300	0.41400	0.29800	0.39300
14	IRNLP	0.51600	0.40200	0.28700	0.45200
15	INSAntive	0.44300	0.38000	0.28100	0.35200
16	NLPPraktikumWS2025	0.44800	0.38500	0.25800	0.33700
17	KostasThesis2025	0.40900	0.31400	0.23900	0.24300
18	NarrativeMiners	0.36900	0.42400	0.23800	0.42600
19	nlptuducd	0.33800	0.36400	0.22600	0.31700
20	ammd7	0.29400	0.35500	0.22200	0.34300
21	UniBonn187	0.40900	0.33000	0.20600	0.22400
22	Irapuarani	0.33500	0.20100	0.18800	0.14100
23	DUTtask10	0.31000	0.40500	0.16500	0.36800
24	LATeIIMAS	0.22200	0.33400	0.16300	0.32300
25	GeorgeSnape	0.19300	0.34800	0.15600	0.34800
26	GrammarPolice	0.16800	0.25800	0.06300	0.12400
27	Baseline	0.03000	0.12700	0.01300	0.07000
28	NarrativeNexus	0.00000	0.00000	0.00000	0.00000
28	bbStar	0.00000	0.00000	0.00000	0.00000

English - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	KyuHyunChoi	0.76686	0.73517	0.75040
2	WordWiz	0.75464	0.73705	0.74551
3	GPLSICORTEX	0.75375	0.73274	0.74280
4	TechSSN	0.73886	0.74568	0.74203
5	NarrativeNexus	0.71991	0.74267	0.73085
6	NarrativeMiners	0.71139	0.74827	0.72910
7	clujteam	0.72350	0.72647	0.72464
8	PATeam	0.72371	0.72589	0.72433
9	TartanTritons	0.72396	0.70200	0.71245
10	YNUzwt	0.69707	0.70112	0.69885
11	LATeIIMAS	0.70540	0.68674	0.69558
12	bbStar	0.66436	0.72023	0.69103
13	Synapse	0.65986	0.69175	0.67527
14	Baseline	0.65144	0.68344	0.66690
15	DUTtask10	0.00000	0.00000	0.00000
15	Mendel292A	0.00000	0.00000	0.00000
15	UMZNLP	0.00000	0.00000	0.00000
15	ftd	0.00000	0.00000	0.00000

Portuguese - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	DUTIR	0.59260	0.64950	0.62540	0.63720	0.94610
2	PATeam	0.49160	0.55370	0.52630	0.53970	0.90570
3	QUST	0.45790	0.50990	0.47680	0.49280	0.84510
4	BERTastic	0.41750	0.47470	0.43650	0.45480	0.84510
5	LTG	0.40740	0.11420	0.11460	0.11440	0.42760
6	DEMON	0.36700	0.43240	0.39630	0.41360	0.80810
7	LATeIIMAS	0.33670	0.38720	0.35600	0.37100	0.71720
8	TartanTritons	0.33330	0.18180	0.16720	0.17420	0.50840
9	gowithnlp	0.26940	0.31650	0.29100	0.30320	0.71380
10	Cimba	0.26260	0.33000	0.30960	0.31950	0.20540
10	FromProblemImportSolve	0.26260	0.32970	0.37150	0.34930	0.65660
11	Fane	0.25590	0.30980	0.28480	0.29680	0.62960
12	Dhananjaya	0.21890	0.26320	0.24770	0.25520	0.76770
13	YNUzwt	0.16160	0.20540	0.18890	0.19680	0.39730
14	HowardUniversityAI4PC	0.13130	0.11320	0.22600	0.15080	0.51850
15	Baseline	0.04710	0.05050	0.04640	0.04840	0.36030
16	teamninebonn	0.00000	0.04040	0.11150	0.05930	0.44110

Portuguese - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	GATENLP	0.66400	0.26000	0.48000	0.25400
2	PATeam	0.54100	0.29000	0.40900	0.26900
3	23	0.48700	0.33800	0.31300	0.25700
4	KostasThesis2025	0.47800	0.20100	0.30900	0.15300
5	iLostTheCode	0.53600	0.27300	0.29300	0.20600
6	Narrlangen	0.48000	0.33900	0.29100	0.25900
7	UNEDTeam	0.53700	0.32400	0.27000	0.26200
8	YNUzwt	0.56800	0.30100	0.26600	0.27700
9	Irapuarani	0.43500	0.19500	0.22500	0.12900
10	INSAntive	0.49100	0.27500	0.21500	0.20400
11	CtrlAltElite	0.49200	0.31600	0.14900	0.21900
12	NotMyNarrative	0.22800	0.25000	0.12400	0.11100
13	DUTtask10	0.10700	0.24100	0.02600	0.13400
14	Baseline	0.03700	0.14000	0.01400	0.07000

Portuguese - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	WordWiz	0.75931	0.73885	0.74857
2	PATeam	0.75365	0.73984	0.74637
3	bbStar	0.70991	0.72924	0.71926
4	YNUzwt	0.68986	0.68603	0.68771
5	TartanTritons	0.69764	0.67419	0.68540
6	Baseline	0.67822	0.68297	0.68046
7	LATeIIMAS	0.68395	0.66339	0.67297
8	DUTtask10	0.00000	0.00000	0.00000

Russian - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	DUTIR	0.56540	0.61750	0.59030	0.60360	0.85510
2	QUST	0.51400	0.56280	0.53300	0.54750	0.83180
3	TartanTritons	0.47200	0.16200	0.15420	0.15800	0.43930
4	BERTastic	0.46730	0.50470	0.47580	0.48980	0.78970
4	DEMON	0.46730	0.51420	0.48020	0.49660	0.81310
5	gowithnlp	0.44860	0.48130	0.45370	0.46710	0.86450
6	PATeam	0.44390	0.49780	0.48900	0.49330	0.78040
7	LTG	0.42990	0.08620	0.08810	0.08710	0.35980
8	Cimba	0.38320	0.41670	0.39650	0.40630	0.33640
9	FromProblemImportSolve	0.35510	0.43700	0.51980	0.47480	0.75230
10	LATeIIMAS	0.31310	0.36040	0.35240	0.35630	0.64490
11	Dhananjaya	0.29440	0.32560	0.30840	0.31670	0.66360
12	YNUzwt	0.26640	0.29910	0.28190	0.29020	0.48600
13	Fane	0.24300	0.27570	0.25990	0.26760	0.60750
14	HowardUniversityAI4PC	0.12620	0.07590	0.12780	0.09520	0.42520
15	Baseline	0.05140	0.06070	0.05730	0.05900	0.34110

Russian - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	GATENLP	0.70900	0.27400	0.51800	0.28200
2	PATeam	0.56600	0.26800	0.43400	0.24700
3	iLostTheCode	0.61800	0.31200	0.41100	0.30800
4	Narrlangen	0.57400	0.37000	0.40500	0.32400
5	YNUzwt	0.50600	0.35100	0.33500	0.30200
6	KostasThesis2025	0.59600	0.25700	0.33300	0.23400
7	UNEDTeam	0.51300	0.32500	0.33000	0.27000
8	INSAntive	0.55400	0.32800	0.32300	0.34200
9	UniBonn187	0.55000	0.30300	0.23100	0.17800
10	Irapuarani	0.35900	0.15000	0.19100	0.09200
11	IRNLP	0.53700	0.35100	0.11600	0.25200
12	GrammarPolice	0.00000	0.00000	0.05000	0.21800
13	DUTtask10	0.04000	0.18500	0.03300	0.18000
14	Baseline	0.06500	0.21300	0.00800	0.06400
15	LATeIIMAS	0.00000	0.00000	0.00000	0.00000

Russian - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	PATeam	0.69984	0.71423	0.70639
2	WordWiz	0.70193	0.70730	0.70395
3	TartanTritons	0.68271	0.68155	0.68159
4	YNUzwt	0.66621	0.68632	0.67578
5	bbStar	0.63433	0.69813	0.66440
6	DUTtask10	0.65979	0.66845	0.66379
7	Baseline	0.62206	0.66872	0.64427
8	LATeIIMAS	0.61043	0.67674	0.64161

Bulgarian - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	PATeam	0.51610	0.53970	0.53120	0.53540	0.92740
2	DUTIR	0.50810	0.53730	0.56250	0.54960	0.94350
3	DEMON	0.45970	0.47970	0.46090	0.47010	0.88710
4	gowithnlp	0.43550	0.45970	0.44530	0.45240	0.83060
5	TartanTritons	0.41130	0.09680	0.09380	0.09520	0.55650
6	QUST	0.38710	0.39200	0.38280	0.38740	0.84680
7	BERTastic	0.35480	0.37100	0.35940	0.36510	0.87100
8	Fane	0.34680	0.36290	0.35160	0.35710	0.77420
9	LTG	0.31450	0.04930	0.05470	0.05190	0.55650
10	Cimba	0.25810	0.26980	0.26560	0.26770	0.25000
11	FromProblemImportSolve	0.20970	0.28660	0.36720	0.32190	0.80650
12	Dhananjaya	0.19350	0.21090	0.21090	0.21090	0.69350
13	HowardUniversityAI4PC	0.09680	0.07920	0.14840	0.10330	0.51610
14	Baseline	0.04030	0.04030	0.03910	0.03970	0.25810
15	TechDacians	0.00000	1.00000	0.00000	0.00000	0.25000

Bulgarian - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	PATeam	0.63100	0.33800	0.46000	0.33300
2	GATENLP	0.62900	0.33200	0.41600	0.28600
3	iLostTheCode	0.55700	0.33500	0.36900	0.30800
4	UNEDTeam	0.57400	0.35300	0.36300	0.31200
5	Narrlangen	0.49500	0.40100	0.35500	0.37600
6	KostasThesis2025	0.51000	0.32200	0.33300	0.30000
7	INSAntive	0.52300	0.36600	0.32400	0.36000
8	Irapuarani	0.36600	0.17700	0.18300	0.12000
9	NotMyNarrative	0.24900	0.30600	0.14200	0.22800
10	DUTtask10	0.17200	0.34100	0.12100	0.31200
11	LATeIIMAS	0.17700	0.26600	0.07200	0.16900
12	Baseline	0.05600	0.19100	0.02200	0.11900

Bulgarian - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	PATeam	0.71405	0.69478	0.70396
2	WordWiz	0.69034	0.67816	0.68390
3	bbStar	0.65751	0.68740	0.67198
4	TartanTritons	0.66758	0.64399	0.65531
5	Baseline	0.62739	0.64180	0.63430
6	LATeIIMAS	0.62947	0.61907	0.62406
7	DUTtask10	0.00000	0.00000	0.00000

Hindi - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	QUST	0.46840	0.59060	0.49480	0.53850	0.76900
2	TartanTritons	0.44620	0.18940	0.15970	0.17330	0.32910
3	BERTastic	0.43990	0.56560	0.47380	0.51570	0.78480
4	DEMON	0.40190	0.52530	0.43460	0.47560	0.75630
5	LTG	0.36390	0.16580	0.15970	0.16270	0.37660
6	Cimba	0.35440	0.45310	0.37960	0.41310	0.47150
7	gowithnlp	0.33540	0.43990	0.36390	0.39830	0.71520
8	DUTIR	0.29430	0.37660	0.31150	0.34100	0.59810
9	Dhananjaya	0.27850	0.37660	0.31150	0.34100	0.70890
10	LATeIIMAS	0.27220	0.37110	0.40310	0.38640	0.63610
11	PATeam	0.26900	0.35330	0.29320	0.32050	0.69620
12	FromProblemImportSolve	0.25630	0.37410	0.39270	0.38310	0.66770
13	Fane	0.23420	0.31650	0.26180	0.28650	0.65510
14	HowardUniversityAI4PC	0.16770	0.11460	0.15180	0.13060	0.35440
15	Baseline	0.05700	0.07910	0.06540	0.07160	0.32280

Hindi - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	DUTtask10	0.56900	0.48400	0.53500	0.49400
2	IRNLP	0.37500	0.46700	0.51500	0.50000
3	Narrlangen	0.39500	0.46400	0.38500	0.46500
4	UNEDTeam	0.44900	0.46000	0.37600	0.45600
5	GATENLP	0.40900	0.41000	0.32100	0.37900
6	KostasThesis2025	0.38400	0.41800	0.28200	0.40200
7	INSAntive	0.36500	0.44000	0.26500	0.41400
8	NotMyNarrative	0.31800	0.44900	0.24300	0.40800
9	PATeam	0.39200	0.39000	0.21800	0.35000
10	iLostTheCode	0.20700	0.31300	0.14700	0.29200
11	Irapuarani	0.23400	0.12500	0.11100	0.08200
12	LATeIIMAS	0.09700	0.18500	0.02900	0.10200
13	Baseline	0.08100	0.26000	0.00000	0.00000
14	bbStar	0.00000	0.00000	0.00000	0.00000

Hindi - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	PATeam	0.75097	0.76045	0.75540
2	WordWiz	0.72859	0.73914	0.73356
3	bbStar	0.71789	0.73704	0.72712
4	TartanTritons	0.69679	0.70119	0.69873
5	Baseline	0.66802	0.67188	0.66974
6	DUTtask10	0.00000	0.00000	0.00000

Leaderboards on the dev set

On December 6th 2024, a hotfix for subtask 1 has been applied to resolve an issue with order-sensitivity in the Exact Match Ratio. Predictions with the same fine-grained labels, regardless of order, are now treated equally.

English - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	nihao	0.59340	0.63440	0.59000	0.61140	0.95600
2	gouge	0.57140	0.61290	0.57000	0.59070	0.95600
4	DEMON	0.53850	0.56520	0.52000	0.54170	0.95600
4	LLL	0.53850	0.61020	0.72000	0.66060	0.92310
5	QUST	0.47250	0.51610	0.48000	0.49740	0.95600
6	gowithnlp	0.43960	0.49450	0.45000	0.47120	0.91210
7	kositeam	0.39560	0.42860	0.39000	0.40840	0.85710
8	Fane	0.38460	0.43960	0.40000	0.41880	0.95600
9	NlpUned	0.37360	0.41760	0.38000	0.39790	0.89010
9	TartanTritons	0.37360	0.17580	0.16000	0.16750	0.71430
10	LATeIIMAS	0.36260	0.39560	0.36000	0.37700	0.86810
11	Blabla	0.34070	0.40220	0.37000	0.38540	0.82420
11	Mekky	0.34070	0.39360	0.37000	0.38140	0.80220
11	MikasaAckerman	0.34070	0.38460	0.35000	0.36650	0.78020
11	NarrativeMiners	0.34070	0.40660	0.37000	0.38740	0.74730
11	kzeky	0.34070	0.37360	0.34000	0.35600	0.84620
12	Agent667	0.32970	0.37250	0.38000	0.37620	0.86810
13	LTG	0.31870	0.07690	0.07000	0.07330	0.78020
13	auriza	0.31870	0.36110	0.39000	0.37500	0.79120
14	AtharvaK	0.30770	0.49330	0.37000	0.42290	0.83520
14	SZM	0.30770	0.43240	0.32000	0.36780	0.89010
14	ToTeamOrNotToTeam	0.30770	0.35160	0.32000	0.33510	0.75820
14	mbzuaijcbcruz	0.30770	0.13040	0.12000	0.12500	0.58240
15	DinDon	0.29670	0.37780	0.34000	0.35790	0.78020
16	Cendekiaaa	0.28570	0.34000	0.34000	0.34000	0.74730
16	alamo	0.28570	0.32320	0.32000	0.32160	0.90110
16	amoriodi	0.28570	0.13190	0.12000	0.12570	0.67030
17	Saeed	0.27470	0.30430	0.28000	0.29170	0.89010
17	m1nadzuki	0.27470	0.29670	0.27000	0.28270	0.89010
18	Malak	0.26370	0.26880	0.25000	0.25910	0.73630
18	Sarfraz307	0.26370	0.27470	0.25000	0.26180	0.70330
18	adithjrajeev	0.26370	0.29900	0.29000	0.29440	0.83520
18	zizo	0.26370	0.30480	0.32000	0.31220	0.81320
19	NarrativeNexus	0.25270	0.30770	0.28000	0.29320	0.71430
19	ShehanMBZUAI	0.25270	0.29290	0.29000	0.29150	0.79120
20	news88readers	0.24180	0.29670	0.27000	0.28270	0.78020
21	YNUzwt	0.23080	0.09780	0.09000	0.09370	0.71430
21	lha	0.23080	0.22950	0.28000	0.25230	0.74730
21	testerbester	0.23080	0.23400	0.22000	0.22680	0.64840
22	Cimba	0.21980	0.25270	0.23000	0.24080	0.09890
22	Dhananjaya	0.21980	0.05490	0.05000	0.05240	0.62640
22	Grimoires	0.21980	0.23160	0.22000	0.22560	0.73630
22	NLP26	0.21980	0.25000	0.25000	0.25000	0.80220
22	Tuebingen	0.21980	0.29290	0.29000	0.29150	0.75820
22	UtkarshAgarwal	0.21980	0.26370	0.24000	0.25130	0.80220
23	KevinMBZUAI	0.20880	0.27370	0.26000	0.26670	0.26370
23	UMZNLP	0.20880	0.22830	0.21000	0.21870	0.69230
24	AMUPD	0.19780	0.24720	0.22000	0.23280	0.83520
24	MeowgicalWitch	0.19780	0.20650	0.19000	0.19790	0.91210
24	Mon	0.19780	0.23810	0.30000	0.26550	0.72530
24	cdos	0.19780	0.24000	0.30000	0.26670	0.65930
25	BoiCoDon	0.17580	0.15450	0.17000	0.16190	0.74730
25	MARK	0.17580	0.19570	0.18000	0.18750	0.62640
25	mosalah	0.17580	0.20880	0.19000	0.19900	0.76920
25	teamninebonn	0.17580	0.21980	0.20000	0.20940	0.72530
26	Harum	0.16480	0.18480	0.17000	0.17710	0.64840
26	IvanRodkin	0.16480	0.18950	0.18000	0.18460	0.72530
26	LPA	0.16480	0.20950	0.22000	0.21460	0.78020
27	Rosetta	0.15380	0.19590	0.19000	0.19290	0.63740
27	arham306	0.15380	0.18480	0.17000	0.17710	0.64840
27	auroracx	0.15380	0.18680	0.17000	0.17800	0.74730
27	hocpocTUCN	0.15380	0.20880	0.19000	0.19900	0.80220
28	Asqii619	0.14290	0.17580	0.16000	0.16750	0.68130
28	Survivors	0.14290	0.15050	0.14000	0.14510	0.80220
28	YNUHPCC	0.14290	0.16480	0.15000	0.15710	0.80220
29	LTX	0.13190	0.14290	0.13000	0.13610	0.80220
29	nnn	0.13190	0.14290	0.13000	0.13610	0.80220
30	Baseline	0.12090	0.12090	0.11000	0.11520	0.80220
30	Kavya	0.12090	0.12090	0.11000	0.11520	0.80220
30	Mendel292A	0.12090	0.12090	0.11000	0.11520	0.80220
30	PAI	0.12090	0.12090	0.11000	0.11520	0.80220
30	Yakkinmookrob	0.12090	0.12090	0.11000	0.11520	0.80220
30	btf	0.12090	0.12090	0.11000	0.11520	0.80220
30	esbay	0.12090	0.12090	0.11000	0.11520	0.80220
30	noob	0.12090	0.12090	0.11000	0.11520	0.80220
31	4oclock	0.10990	0.26730	0.27000	0.26870	0.69230
32	HowardUniversityAI4PC	0.08790	0.07010	0.11000	0.08560	0.72530
33	Team12	0.07690	0.06250	0.06000	0.06120	0.75820
34	Synapse	0.06590	0.25730	0.44000	0.32470	0.92310
35	SavvyScout	0.05490	0.11670	0.30000	0.16810	0.82420
35	bumblebeeTransformer	0.05490	0.15110	0.21000	0.17570	0.74730
36	Gzxy	0.04400	0.04400	0.04000	0.04190	0.14290
37	CAISA	0.03300	0.03300	0.03000	0.03140	0.78020
38	cocoa	0.02200	0.10220	0.14000	0.11810	0.80220
39	Besher	0.00000	0.00000	0.00000	0.00000	0.00000
39	Bilal	0.00000	0.00000	0.00000	0.00000	0.00000
39	Genadium	0.00000	0.00000	0.00000	0.00000	0.00000
40	GuitarBaseline	0.00000	0.05450	0.40000	0.09590	0.80220
41	Irapuarani	0.00000	0.00000	0.00000	0.00000	0.00000
41	M2024A	0.00000	0.00000	0.00000	0.00000	0.00000
41	MME	0.00000	0.00000	0.00000	0.00000	0.00000
41	MyriamB	0.00000	0.00000	0.00000	0.00000	0.00000
41	NLPPraktikumWS2025	0.00000	0.00000	0.00000	0.00000	0.00000
41	RachelBakerPalen22	0.00000	0.00000	0.00000	0.00000	0.00000
42	TechDacians	0.00000	1.00000	0.00000	0.00000	0.09890
43	iLostTheCode	0.00000	0.00000	0.00000	0.00000	0.00000
43	insun606	0.00000	0.00000	0.00000	0.00000	0.00000
43	telorbulat	0.00000	0.00000	0.00000	0.00000	0.00000

English - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	COGNAC	0.59600	0.38700	0.54700	0.39000
2	iLostTheCode	0.65000	0.33200	0.52000	0.35300
3	GATENLP	0.60300	0.38400	0.51700	0.37700
4	INSALyon2	0.53700	0.35600	0.49200	0.38300
5	Team19	0.60200	0.36100	0.45800	0.38600
7	UNEDTeam	0.51200	0.34500	0.38400	0.34100
8	23	0.49300	0.40400	0.37100	0.40700
9	KostasThesis2025	0.51500	0.33600	0.35100	0.32600
10	Narrlangen	0.40200	0.39900	0.34900	0.38800
11	ammd7	0.26800	0.44300	0.33700	0.39500
12	Apfelkuchen	0.39200	0.38400	0.33600	0.42000
13	ftd	0.43600	0.33400	0.33000	0.41200
14	NotMyNarrative	0.43200	0.43800	0.32500	0.41700
15	GrammarPolice	0.17100	0.37600	0.31800	0.32900
15	NLPPraktikumWS2025	0.45200	0.41300	0.31800	0.37600
16	CtrlAltElite	0.42600	0.42500	0.31400	0.43300
17	GeorgeSnape	0.24800	0.41800	0.28300	0.43900
18	Abdulaziz	0.26800	0.44300	0.26800	0.44300
18	CodeExperts	0.26800	0.44300	0.26800	0.44300
18	Dhabi	0.19500	0.39600	0.26800	0.44300
18	EGwal1A	0.26800	0.44300	0.26800	0.44300
18	IRNLP	0.22300	0.37500	0.26800	0.44300
18	Irapuarani	0.26800	0.44300	0.26800	0.44300
18	NCLteam	0.26800	0.44300	0.26800	0.44300
18	RachelBakerPalen22	0.19400	0.21300	0.26800	0.44300
18	m1nadzuki	0.26800	0.44300	0.26800	0.44300
19	Alaa	0.22300	0.34800	0.18800	0.37700
20	nlptuducd	0.26200	0.33000	0.18600	0.28200
21	UniBonn187	0.37400	0.31400	0.17300	0.20500
22	Blabla	0.19600	0.35000	0.14000	0.33200
23	YNUzwt	0.04900	0.21500	0.12200	0.32700
24	DUTtask10	0.14100	0.33200	0.09800	0.29700
24	NarrativeMiners	0.09800	0.29700	0.09800	0.29700
24	TransformerTribe	0.21900	0.32800	0.09800	0.29700
25	CoSmOS	0.12800	0.24600	0.07900	0.18400
26	DMCT	0.18100	0.33000	0.07300	0.26000
27	belta8	0.22600	0.27600	0.06700	0.18100
28	BaSingSe	0.10800	0.24200	0.05600	0.21800
29	Cipher	0.34300	0.38200	0.04900	0.12500
30	PAI	0.03500	0.13300	0.01400	0.06600
31	Asqii619	0.00000	0.00000	0.00000	0.00000
32	Baseline	0.10600	0.26700	0.00000	0.00000
33	Malak	0.00000	0.00000	0.00000	0.00000
33	NarrativeNexus	0.00000	0.00000	0.00000	0.00000
34	NlpUned	0.00000	0.00000	0.00000	0.00000
34	news88readers	0.10600	0.26700	0.00000	0.00000

English - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	NarrativeMiners	0.79489	0.83321	0.81339
3	WordWiz	0.76589	0.73367	0.74905
4	GPLSICORTEX	0.76514	0.73193	0.74798
5	DAVICHI	0.76044	0.73454	0.74699
6	Besher	0.75910	0.73481	0.74662
7	Edd	0.75922	0.73416	0.74607
8	MME	0.78395	0.71090	0.74533
9	Bilal	0.75648	0.73447	0.74517
10	TechSSN	0.74648	0.74406	0.74512
11	Genadium	0.75398	0.73138	0.74233
12	notmena	0.75088	0.72337	0.73661
13	telorbulat	0.73944	0.72474	0.73181
14	NarrativeNexus	0.71796	0.73931	0.72839
15	OmarElHerraoui	0.73826	0.71351	0.72543
16	SholpanBolatzhanova	0.72479	0.72540	0.72494
17	clujteam	0.72232	0.71393	0.71801
18	insun606	0.72366	0.70875	0.71595
19	Synapse	0.72342	0.70785	0.71536
20	m1nadzuki	0.72482	0.70139	0.71260
21	TartanTritons	0.72603	0.69625	0.71058
22	DUTtask10	0.72734	0.69426	0.71016
23	honey	0.71193	0.69861	0.70506
24	Kavya	0.76003	0.65646	0.70438
26	LATeIIMAS	0.71738	0.68822	0.70219
27	MyriamB	0.70125	0.70137	0.70114
28	amoriodi	0.68856	0.71412	0.70103
29	YNUzwt	0.70214	0.69439	0.69796
30	bbStar	0.66883	0.71327	0.69026
31	UMZNLP	0.66495	0.68985	0.67712
32	AJ805	0.65667	0.68130	0.66871
33	Baseline	0.65540	0.67957	0.66719
33	YNUHPCC	0.65540	0.67957	0.66719
33	news88readers	0.65540	0.67957	0.66719
34	Dhananjaya	0.00000	0.00000	0.00000
34	M2024A	0.00000	0.00000	0.00000
34	hacker	0.00000	0.00000	0.00000

Portuguese - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
2	QUST	0.74140	0.77500	0.75000	0.76230	0.91380
2	nihao	0.74140	0.79830	0.76610	0.78190	0.93970
3	LTG	0.71550	0.26500	0.25000	0.25730	0.37930
4	DEMON	0.70690	0.76720	0.71770	0.74170	0.96550
5	TartanTritons	0.69830	0.34480	0.32260	0.33330	0.43970
6	Blabla	0.66380	0.77480	0.69350	0.73190	0.94830
7	Cimba	0.65520	0.69230	0.65320	0.67220	0.25000
7	LATeIIMAS	0.65520	0.69830	0.65320	0.67500	0.87930
8	LPA	0.62930	0.67740	0.67740	0.67740	0.85340
8	lha	0.62930	0.67230	0.64520	0.65840	0.85340
9	Dhananjaya	0.62070	0.38980	0.37100	0.38020	0.52590
10	Fane	0.59480	0.64660	0.60480	0.62500	0.76720
10	Grimoires	0.59480	0.62810	0.61290	0.62040	0.83620
11	Sarfraz307	0.56900	0.59320	0.56450	0.57850	0.68970
11	gowithnlp	0.56900	0.60870	0.56450	0.58580	0.82760
12	ToTeamOrNotToTeam	0.55170	0.57760	0.54030	0.55830	0.75000
13	teamninebonn	0.52590	0.55170	0.51610	0.53330	0.62930
14	Baseline	0.44830	0.48280	0.45160	0.46670	0.24140
15	amoriodi	0.42240	0.18100	0.16940	0.17500	0.26720
16	YNUzwt	0.31900	0.24140	0.22580	0.23330	0.28450
17	kzeky	0.27590	0.29660	0.28230	0.28930	0.46550
18	Team12	0.22410	0.10880	0.16940	0.13250	0.27590
19	Malak	0.00000	0.00000	0.00000	0.00000	0.25860
19	TechDacians	0.00000	1.00000	0.00000	0.00000	0.25000

Portuguese - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	iLostTheCode	0.70500	0.27100	0.55700	0.27100
2	23	0.74800	0.24700	0.53800	0.28000
3	ftd	0.71300	0.30400	0.52100	0.27400
4	KostasThesis2025	0.67800	0.25100	0.47100	0.23700
5	Apfelkuchen	0.67000	0.29500	0.43400	0.38000
6	Narrlangen	0.61800	0.35200	0.42800	0.33600
7	Team19	0.62400	0.25800	0.36800	0.35300
8	CtrlAltElite	0.70600	0.30600	0.36400	0.38300
10	NotMyNarrative	0.44200	0.40000	0.27000	0.29100
11	UNEDTeam	0.44000	0.40000	0.25000	0.30800
12	Irapuarani	0.37100	0.13300	0.22800	0.09000
13	EGwal1A	0.37400	0.18600	0.20500	0.15600
14	GrammarPolice	0.03300	0.13700	0.03800	0.17400
15	DUTtask10	0.04300	0.18400	0.02900	0.16700
15	IRNLP	0.35400	0.41200	0.02900	0.16700
15	TransformerTribe	0.64200	0.37800	0.02900	0.16700
15	YNUzwt	0.00000	0.00000	0.02900	0.16700
16	Baseline	0.06700	0.18800	0.01000	0.05600
17	DMCT	0.05400	0.19300	0.00000	0.00000

Portuguese - Subtask 3

Rank	Team	Precision	Recall	F1 macro
2	WordWiz	0.74135	0.74993	0.74521
3	telorbulat	0.75258	0.72481	0.73806
4	OmarElHerraoui	0.72418	0.73771	0.73058
5	amoriodi	0.71370	0.73489	0.72393
6	TartanTritons	0.69592	0.69489	0.69517
7	YNUzwt	0.68247	0.69420	0.68810
8	LATeIIMAS	0.67980	0.68359	0.68123
9	DUTtask10	0.68290	0.66820	0.67529
10	Baseline	0.65332	0.67644	0.66453

Hindi - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	TartanTritons	0.48570	0.12280	0.11360	0.11800	0.35360
2	DEMON	0.47500	0.54870	0.49350	0.51970	0.79290
3	Blabla	0.47140	0.56620	0.50000	0.53100	0.78930
4	LTG	0.41070	0.15000	0.13640	0.14290	0.38570
5	nihao	0.39640	0.44130	0.40260	0.42110	0.66790
6	Cimba	0.39290	0.46100	0.42210	0.44070	0.41790
7	QUST	0.38570	0.46180	0.43180	0.44630	0.71430
8	Fane	0.32140	0.35710	0.32470	0.34010	0.68570
9	LPA	0.29640	0.35530	0.36690	0.36100	0.63570
10	lha	0.28930	0.35080	0.37010	0.36020	0.58570
11	Dhananjaya	0.24290	0.11390	0.10390	0.10870	0.42140
12	kzeky	0.23570	0.26690	0.26950	0.26820	0.55000
13	LATeIIMAS	0.18570	0.26190	0.32140	0.28860	0.57860
14	Baseline	0.17140	0.18930	0.17210	0.18030	0.34290
15	gowithnlp	0.13210	0.23230	0.34090	0.27630	0.54290
16	Malak	0.01790	0.05290	0.02920	0.03770	0.37140

Hindi - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	GATENLP	0.51300	0.41100	0.41500	0.40300
2	KostasThesis2025	0.49400	0.34100	0.37500	0.32100
3	ftd	0.37800	0.33800	0.33000	0.36000
4	Narrlangen	0.45400	0.39100	0.31400	0.36700
5	Team19	0.35700	0.31700	0.30600	0.33600
7	UNEDTeam	0.31400	0.38700	0.22400	0.34600
8	Irapuarani	0.32000	0.15600	0.17300	0.09000
9	NotMyNarrative	0.23800	0.35400	0.16800	0.31500
10	iLostTheCode	0.23400	0.33500	0.14900	0.27100
11	Apfelkuchen	0.34500	0.40000	0.14300	0.32200
12	DUTtask10	0.05700	0.23200	0.05700	0.23200
12	IRNLP	0.06700	0.15200	0.05700	0.19900
13	Baseline	0.10000	0.26200	0.05100	0.18700

Hindi - Subtask 3

Rank	Team	Precision	Recall	F1 macro
2	WordWiz	0.73767	0.73789	0.73745
3	bbStar	0.72153	0.73394	0.72740
4	amoriodi	0.72694	0.69082	0.70803
5	TartanTritons	0.70364	0.69667	0.69959
6	HitManZombie	0.68260	0.69997	0.69098
7	telorbulat	0.69762	0.67638	0.68616
8	OmarElHerraoui	0.69865	0.66637	0.68181
9	Baseline	0.67102	0.66892	0.66916

Bulgarian - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
2	QUST	0.54840	0.57580	0.55880	0.56720	0.70970
3	nihao	0.48390	0.58060	0.52940	0.55380	0.74190
4	DEMON	0.41940	0.51610	0.47060	0.49230	0.74190
4	TartanTritons	0.41940	0.09380	0.08820	0.09090	0.51610
5	LTG	0.38710	0.17140	0.17650	0.17390	0.54840
6	Cimba	0.32260	0.35480	0.32350	0.33850	0.25810
6	Dhananjaya	0.32260	0.18180	0.17650	0.17910	0.58060
7	gowithnlp	0.29030	0.35480	0.32350	0.33850	0.87100
8	Blabla	0.25810	0.40000	0.41180	0.40580	0.74190
8	Fane	0.25810	0.35480	0.32350	0.33850	0.83870
9	LPA	0.22580	0.35710	0.44120	0.39470	0.90320
9	lha	0.22580	0.36110	0.38240	0.37140	0.77420
10	LATeIIMAS	0.19350	0.19440	0.20590	0.20000	0.67740
10	kzeky	0.19350	0.21880	0.20590	0.21210	0.54840
11	HowardUniversityAI4PC	0.16130	0.17780	0.23530	0.20250	0.45160
12	Baseline	0.06450	0.06450	0.05880	0.06150	0.64520
13	Malak	0.00000	0.00000	0.00000	0.00000	0.09680
13	TechDacians	0.00000	1.00000	0.00000	0.00000	0.25810

Bulgarian - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
2	Narrlangen	0.50500	0.42300	0.42000	0.39800
3	KostasThesis2025	0.61900	0.36200	0.39600	0.33100
4	UNEDTeam	0.47800	0.40600	0.38500	0.39100
5	NotMyNarrative	0.40300	0.40200	0.31500	0.39600
6	Apfelkuchen	0.52600	0.42900	0.30000	0.41200
7	ftd	0.41100	0.42200	0.29000	0.38300
8	Team19	0.34700	0.40400	0.19000	0.28500
9	iLostTheCode	0.22400	0.36700	0.18800	0.31400
10	Irapuarani	0.34700	0.15400	0.18600	0.10500
11	DUTtask10	0.15200	0.35000	0.15600	0.35300
12	Baseline	0.03800	0.15500	0.01400	0.08300
13	IRNLP	0.21000	0.39100	0.00700	0.04200

Bulgarian - Subtask 3

Rank	Team	Precision	Recall	F1 macro
2	OmarElHerraoui	0.69936	0.71289	0.70572
3	amoriodi	0.68162	0.72907	0.70430
4	WordWiz	0.68636	0.70079	0.69296
5	telorbulat	0.67677	0.67249	0.67409
6	TartanTritons	0.67444	0.67217	0.67302
7	LATeIIMAS	0.63729	0.64177	0.63938
8	Baseline	0.61946	0.65360	0.63589

Russian - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
2	nihao	0.58140	0.59770	0.58430	0.59090	0.88370
3	QUST	0.55810	0.57950	0.57300	0.57630	0.89530
4	TartanTritons	0.54650	0.11630	0.11240	0.11430	0.47670
4	gowithnlp	0.54650	0.58140	0.56180	0.57140	0.90700
5	DEMON	0.53490	0.54650	0.52810	0.53710	0.89530
6	LTG	0.46510	0.11830	0.12360	0.12090	0.53490
7	Dhananjaya	0.39530	0.40700	0.39330	0.40000	0.69770
8	Cimba	0.38370	0.38370	0.37080	0.37710	0.32560
9	Fane	0.32560	0.34880	0.33710	0.34290	0.73260
10	YNUzwt	0.30230	0.15120	0.14610	0.14860	0.48840
11	LATeIIMAS	0.25580	0.28090	0.28090	0.28090	0.66280
12	Mendel292A	0.23260	0.20930	0.20220	0.20570	0.56980
13	Baseline	0.05810	0.05810	0.05620	0.05710	0.33720

Russian - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
2	UNEDTeam	0.61100	0.37500	0.41100	0.36900
3	Narrlangen	0.53600	0.35700	0.34800	0.33700
4	KostasThesis2025	0.48300	0.25800	0.27700	0.20700
5	Apfelkuchen	0.51600	0.31900	0.24800	0.37000
6	ftd	0.51900	0.31600	0.23500	0.31800
7	iLostTheCode	0.42800	0.26000	0.23400	0.17300
8	Irapuarani	0.35400	0.17600	0.17400	0.09900
9	UniBonn187	0.52200	0.30000	0.17100	0.11800
10	NotMyNarrative	0.24800	0.34400	0.16800	0.33100
11	Team19	0.54000	0.32900	0.15700	0.22700
12	DUTtask10	0.14600	0.34300	0.14600	0.34300
13	IRNLP	0.44200	0.31300	0.13100	0.30600
14	GrammarPolice	0.00000	0.00000	0.12500	0.33100
15	TransformerTribe	0.38700	0.39700	0.06200	0.24200
16	YNUzwt	0.03100	0.17400	0.03100	0.17400
17	Baseline	0.04100	0.13800	0.01300	0.07000
18	belta8	0.00000	0.00000	0.00000	0.00000

Russian - Subtask 3

Rank	Team	Precision	Recall	F1 macro
2	WordWiz	0.71180	0.71331	0.71217
3	TartanTritons	0.68762	0.67250	0.67971
4	YNUzwt	0.66752	0.68006	0.67346
5	LATeIIMAS	0.61510	0.67831	0.64490
6	Baseline	0.62504	0.66336	0.64332

Note that the leaderboard is updated every 20 minutes, therefore your submission may not be listed right away.

Contact

twitter.com/_proppy_
Register to the google group of the task to ask questions and interact with other participants

SemEval 2025 shared task on "Multilingual Characterization and Extraction of Narratives from Online News"

Leaderboard

Official SemEval Results on the Test Set

English - Subtask 1

English - Subtask 2

English - Subtask 3

Portuguese - Subtask 1

Portuguese - Subtask 2

Portuguese - Subtask 3

Russian - Subtask 1

Russian - Subtask 2

Russian - Subtask 3

Bulgarian - Subtask 1

Bulgarian - Subtask 2

Bulgarian - Subtask 3

Hindi - Subtask 1

Hindi - Subtask 2

Hindi - Subtask 3

Leaderboards on the dev set

English - Subtask 1

English - Subtask 2

English - Subtask 3

Portuguese - Subtask 1

Portuguese - Subtask 2

Portuguese - Subtask 3

Hindi - Subtask 1

Hindi - Subtask 2

Hindi - Subtask 3

Bulgarian - Subtask 1

Bulgarian - Subtask 2

Bulgarian - Subtask 3

Russian - Subtask 1

Russian - Subtask 2

Russian - Subtask 3

Contact