Leaderboard

On December 6th 2024, a hotfix for subtask 1 has been applied to resolve an issue with order-sensitivity in the Exact Match Ratio. Predictions with the same fine-grained labels, regardless of order, are now treated equally.

English - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	nihao	0.59340	0.63440	0.59000	0.61140	0.95600
2	gouge	0.57140	0.61290	0.57000	0.59070	0.95600
3	pateam	0.54950	0.60640	0.57000	0.58760	0.92310
4	DEMON	0.53850	0.56520	0.52000	0.54170	0.95600
4	LLL	0.53850	0.61020	0.72000	0.66060	0.92310
5	QUST	0.47250	0.51610	0.48000	0.49740	0.95600
6	gowithnlp	0.43960	0.49450	0.45000	0.47120	0.91210
7	kositeam	0.39560	0.42860	0.39000	0.40840	0.85710
8	LATeIIMAS	0.35160	0.39800	0.39000	0.39390	0.86810
8	NlpUned	0.35160	0.38460	0.35000	0.36650	0.92310
9	Blabla	0.34070	0.40220	0.37000	0.38540	0.82420
9	Mekky	0.34070	0.39360	0.37000	0.38140	0.80220
9	MikasaAckerman	0.34070	0.38460	0.35000	0.36650	0.78020
9	kzeky	0.34070	0.37360	0.34000	0.35600	0.84620
10	Agent667	0.32970	0.37250	0.38000	0.37620	0.86810
11	LTG	0.31870	0.07690	0.07000	0.07330	0.78020
11	auriza	0.31870	0.36110	0.39000	0.37500	0.79120
12	AtharvaK	0.30770	0.49330	0.37000	0.42290	0.83520
12	SZM	0.30770	0.43240	0.32000	0.36780	0.89010
12	ToTeamOrNotToTeam	0.30770	0.35160	0.32000	0.33510	0.75820
12	mbzuaijcbcruz	0.30770	0.13040	0.12000	0.12500	0.58240
13	DinDon	0.29670	0.37780	0.34000	0.35790	0.78020
13	NarrativeMiners	0.29670	0.30770	0.28000	0.29320	0.80220
14	Cendekiaaa	0.28570	0.34000	0.34000	0.34000	0.74730
14	alamo	0.28570	0.32320	0.32000	0.32160	0.90110
14	amoriodi	0.28570	0.13190	0.12000	0.12570	0.67030
15	Saeed	0.27470	0.30430	0.28000	0.29170	0.89010
15	m1nadzuki	0.27470	0.29670	0.27000	0.28270	0.89010
16	Malak	0.26370	0.26880	0.25000	0.25910	0.73630
16	Sarfraz307	0.26370	0.27470	0.25000	0.26180	0.70330
16	adithjrajeev	0.26370	0.29900	0.29000	0.29440	0.83520
16	zizo	0.26370	0.30480	0.32000	0.31220	0.81320
17	ShehanMBZUAI	0.25270	0.29290	0.29000	0.29150	0.79120
18	news88readers	0.24180	0.29670	0.27000	0.28270	0.78020
19	YNUzwt	0.23080	0.09780	0.09000	0.09370	0.71430
19	lha	0.23080	0.22950	0.28000	0.25230	0.74730
19	testerbester	0.23080	0.23400	0.22000	0.22680	0.64840
20	Dhananjaya	0.21980	0.05490	0.05000	0.05240	0.62640
20	Grimoires	0.21980	0.23160	0.22000	0.22560	0.73630
20	NLP26	0.21980	0.25000	0.25000	0.25000	0.80220
20	Tuebingen	0.21980	0.29290	0.29000	0.29150	0.75820
20	UtkarshAgarwal	0.21980	0.26370	0.24000	0.25130	0.80220
21	KevinMBZUAI	0.20880	0.27370	0.26000	0.26670	0.26370
22	AMUPD	0.19780	0.24720	0.22000	0.23280	0.83520
22	Fane	0.19780	0.21980	0.20000	0.20940	0.67030
22	MeowgicalWitch	0.19780	0.20650	0.19000	0.19790	0.91210
22	Mon	0.19780	0.23810	0.30000	0.26550	0.72530
22	cdos	0.19780	0.24000	0.30000	0.26670	0.65930
23	BoiCoDon	0.17580	0.15450	0.17000	0.16190	0.74730
23	MARK	0.17580	0.19570	0.18000	0.18750	0.62640
23	mosalah	0.17580	0.20880	0.19000	0.19900	0.76920
24	Harum	0.16480	0.18480	0.17000	0.17710	0.64840
24	IvanRodkin	0.16480	0.18950	0.18000	0.18460	0.72530
24	LPA	0.16480	0.20950	0.22000	0.21460	0.78020
25	arham306	0.15380	0.18480	0.17000	0.17710	0.64840
25	auroracx	0.15380	0.18680	0.17000	0.17800	0.74730
25	hocpocTUCN	0.15380	0.20880	0.19000	0.19900	0.80220
26	Asqii619	0.14290	0.17580	0.16000	0.16750	0.68130
26	Survivors	0.14290	0.15050	0.14000	0.14510	0.80220
26	YNUHPCC	0.14290	0.16480	0.15000	0.15710	0.80220
27	Cimba	0.13190	0.15380	0.14000	0.14660	0.80220
27	LTX	0.13190	0.14290	0.13000	0.13610	0.80220
27	nnn	0.13190	0.14290	0.13000	0.13610	0.80220
28	Baseline	0.12090	0.12090	0.11000	0.11520	0.80220
28	Kavya	0.12090	0.12090	0.11000	0.11520	0.80220
28	PAI	0.12090	0.12090	0.11000	0.11520	0.80220
28	Yakkinmookrob	0.12090	0.12090	0.11000	0.11520	0.80220
28	btf	0.12090	0.12090	0.11000	0.11520	0.80220
28	esbay	0.12090	0.12090	0.11000	0.11520	0.80220
28	noob	0.12090	0.12090	0.11000	0.11520	0.80220
29	4oclock	0.10990	0.26730	0.27000	0.26870	0.69230
30	Synapse	0.06590	0.25730	0.44000	0.32470	0.92310
31	SavvyScout	0.05490	0.11670	0.30000	0.16810	0.82420
32	Gzxy	0.04400	0.04400	0.04000	0.04190	0.14290
33	CAISA	0.03300	0.03300	0.03000	0.03140	0.78020
34	Besher	0.00000	0.00000	0.00000	0.00000	0.00000
34	Bilal	0.00000	0.00000	0.00000	0.00000	0.00000
34	Genadium	0.00000	0.00000	0.00000	0.00000	0.00000
35	GuitarBaseline	0.00000	1.00000	0.00000	0.00000	0.80220
36	Irapuarani	0.00000	0.00000	0.00000	0.00000	0.00000
36	M2024A	0.00000	0.00000	0.00000	0.00000	0.00000
36	MME	0.00000	0.00000	0.00000	0.00000	0.00000
36	MyriamB	0.00000	0.00000	0.00000	0.00000	0.00000
36	NLPPraktikumWS2025	0.00000	0.00000	0.00000	0.00000	0.00000
37	NarrativeNexus	0.00000	1.00000	0.00000	0.00000	0.76920
38	RachelBakerPalen22	0.00000	0.00000	0.00000	0.00000	0.00000
39	Team12	0.00000	0.08240	0.15000	0.10640	0.72530
40	iLostTheCode	0.00000	0.00000	0.00000	0.00000	0.00000
40	insun606	0.00000	0.00000	0.00000	0.00000	0.00000
40	telorbulat	0.00000	0.00000	0.00000	0.00000	0.00000

English - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	ammd7	1.00000	0.00000	1.00000	0.00000
2	GATENLP	0.63100	0.37600	0.45500	0.39100
3	pateam	0.63100	0.34200	0.44600	0.34400
4	UNEDTeam	0.47000	0.37100	0.33000	0.31600
5	KostasThesis2025	0.45400	0.37200	0.31800	0.31000
6	GeorgeSnape	0.24800	0.41800	0.28300	0.43900
7	Apfelkuchen	0.43300	0.37400	0.27800	0.44100
8	NotMyNarrative	0.28200	0.44300	0.27600	0.44100
9	Abdulaziz	0.26800	0.44300	0.26800	0.44300
9	CodeExperts	0.26800	0.44300	0.26800	0.44300
9	Dhabi	0.19500	0.39600	0.26800	0.44300
9	IRNLP	0.22300	0.37500	0.26800	0.44300
9	Irapuarani	0.26800	0.44300	0.26800	0.44300
9	NCLteam	0.26800	0.44300	0.26800	0.44300
9	NLPPraktikumWS2025	0.26800	0.44300	0.26800	0.44300
9	RachelBakerPalen22	0.19400	0.21300	0.26800	0.44300
9	iLostTheCode	0.26800	0.44300	0.26800	0.44300
9	m1nadzuki	0.26800	0.44300	0.26800	0.44300
10	EGwal1A	0.19500	0.39600	0.19500	0.39600
11	Alaa	0.22300	0.34800	0.18800	0.37700
12	Blabla	0.19600	0.35000	0.14000	0.33200
13	YNUzwt	0.04900	0.21500	0.12200	0.32700
14	DUTtask10	0.14100	0.33200	0.09800	0.29700
14	NarrativeMiners	0.09800	0.29700	0.09800	0.29700
15	CoSmOS	0.12800	0.24600	0.07900	0.18400
16	Narrlangen	0.26400	0.33100	0.07300	0.19000
16	TransformerTribe	0.11800	0.29500	0.07300	0.26000
17	belta8	0.22600	0.27600	0.06700	0.18100
18	BaSingSe	0.10800	0.24200	0.05600	0.21800
19	PAI	0.03500	0.13300	0.01400	0.06600
20	Asqii619	0.00000	0.00000	0.00000	0.00000
21	Baseline	0.10600	0.26700	0.00000	0.00000
22	Malak	0.00000	0.00000	0.00000	0.00000
23	NlpUned	0.00000	0.00000	0.00000	0.00000
23	news88readers	0.10600	0.26700	0.00000	0.00000

English - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	GPLSICORTEX	0.76514	0.73193	0.74798
2	DAVICHI	0.76044	0.73454	0.74699
3	Besher	0.75910	0.73481	0.74662
4	Edd	0.75922	0.73416	0.74607
5	MME	0.78395	0.71090	0.74533
6	Bilal	0.75648	0.73447	0.74517
7	Genadium	0.75398	0.73138	0.74233
8	NarrativeMiners	0.72860	0.74883	0.73848
9	notmena	0.75088	0.72337	0.73661
10	telorbulat	0.73944	0.72474	0.73181
11	OmarElHerraoui	0.73826	0.71351	0.72543
12	SholpanBolatzhanova	0.72479	0.72540	0.72494
13	insun606	0.72366	0.70875	0.71595
14	Synapse	0.72342	0.70785	0.71536
15	WordWiz	0.71390	0.71481	0.71426
16	m1nadzuki	0.72482	0.70139	0.71260
17	DUTtask10	0.72734	0.69426	0.71016
18	honey	0.71193	0.69861	0.70506
19	Kavya	0.76003	0.65646	0.70438
20	MyriamB	0.70125	0.70137	0.70114
21	amoriodi	0.68856	0.71412	0.70103
22	LATeIIMAS	0.70139	0.68751	0.69404
23	YNUzwt	0.69490	0.68173	0.68749
24	AJ805	0.65667	0.68130	0.66871
25	Baseline	0.65540	0.67957	0.66719
25	YNUHPCC	0.65540	0.67957	0.66719
25	news88readers	0.65540	0.67957	0.66719
26	Dhananjaya	0.00000	0.00000	0.00000
26	M2024A	0.00000	0.00000	0.00000
26	NarrativeNexus	0.00000	0.00000	0.00000
26	hacker	0.00000	0.00000	0.00000

Portuguese - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	pateam	0.77590	0.84620	0.79840	0.82160	0.98280
2	QUST	0.74140	0.77500	0.75000	0.76230	0.91380
2	nihao	0.74140	0.79830	0.76610	0.78190	0.93970
3	LTG	0.71550	0.26500	0.25000	0.25730	0.37930
4	DEMON	0.70690	0.76720	0.71770	0.74170	0.96550
5	Blabla	0.66380	0.77480	0.69350	0.73190	0.94830
6	LPA	0.62930	0.67740	0.67740	0.67740	0.85340
6	lha	0.62930	0.67230	0.64520	0.65840	0.85340
7	Dhananjaya	0.62070	0.38980	0.37100	0.38020	0.52590
8	LATeIIMAS	0.60340	0.61670	0.59680	0.60660	0.84480
9	Grimoires	0.59480	0.62810	0.61290	0.62040	0.83620
10	Sarfraz307	0.56900	0.59320	0.56450	0.57850	0.68970
10	gowithnlp	0.56900	0.60870	0.56450	0.58580	0.82760
11	ToTeamOrNotToTeam	0.55170	0.57760	0.54030	0.55830	0.75000
12	Baseline	0.44830	0.48280	0.45160	0.46670	0.24140
13	amoriodi	0.42240	0.18100	0.16940	0.17500	0.26720
14	YNUzwt	0.31900	0.24140	0.22580	0.23330	0.28450
15	kzeky	0.27590	0.29660	0.28230	0.28930	0.46550
16	Malak	0.00000	0.00000	0.00000	0.00000	0.25860

Portuguese - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	iLostTheCode	0.70500	0.27100	0.55700	0.27100
2	Apfelkuchen	0.67000	0.29500	0.43400	0.38000
3	pateam	0.69000	0.32300	0.32500	0.31100
4	UNEDTeam	0.44000	0.40000	0.25000	0.30800
5	NotMyNarrative	0.33300	0.37100	0.22500	0.30400
6	Irapuarani	0.46600	0.32300	0.03200	0.10800
7	IRNLP	0.35400	0.41200	0.02900	0.16700
7	KostasThesis2025	0.02900	0.16700	0.02900	0.16700
7	YNUzwt	0.00000	0.00000	0.02900	0.16700
8	Baseline	0.06700	0.18800	0.01000	0.05600

Portuguese - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	telorbulat	0.75258	0.72481	0.73806
2	OmarElHerraoui	0.72418	0.73771	0.73058
3	amoriodi	0.71370	0.73489	0.72393
4	DUTtask10	0.68290	0.66820	0.67529
5	YNUzwt	0.67058	0.66975	0.66943
6	Baseline	0.65332	0.67644	0.66453

Hindi - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	pateam	0.48570	0.56030	0.51300	0.53560	0.80360
2	DEMON	0.47500	0.54870	0.49350	0.51970	0.79290
3	Blabla	0.47140	0.56620	0.50000	0.53100	0.78930
4	LTG	0.41070	0.15000	0.13640	0.14290	0.38570
5	nihao	0.39640	0.44130	0.40260	0.42110	0.66790
6	QUST	0.38570	0.46180	0.43180	0.44630	0.71430
7	LPA	0.29640	0.35530	0.36690	0.36100	0.63570
8	lha	0.28930	0.35080	0.37010	0.36020	0.58570
9	Dhananjaya	0.24290	0.11390	0.10390	0.10870	0.42140
10	kzeky	0.23570	0.26690	0.26950	0.26820	0.55000
11	Baseline	0.17140	0.18930	0.17210	0.18030	0.34290
12	gowithnlp	0.13210	0.23230	0.34090	0.27630	0.54290
13	Malak	0.01790	0.05290	0.02920	0.03770	0.37140

Hindi - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	pateam	0.45600	0.27700	0.25800	0.21300
2	UNEDTeam	0.31400	0.38700	0.22400	0.34600
3	iLostTheCode	0.23400	0.33500	0.14900	0.27100
4	Apfelkuchen	0.34500	0.40000	0.14300	0.32200
5	GATENLP	0.14700	0.29100	0.09900	0.25100
6	NotMyNarrative	0.13100	0.25600	0.09500	0.23800
7	IRNLP	0.06700	0.15200	0.05700	0.19900
8	Baseline	0.10000	0.26200	0.05100	0.18700
9	Irapuarani	0.02900	0.16700	0.02900	0.16700

Hindi - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	amoriodi	0.72694	0.69082	0.70803
2	HitManZombie	0.68260	0.69997	0.69098
3	telorbulat	0.69762	0.67638	0.68616
4	OmarElHerraoui	0.69865	0.66637	0.68181
5	Baseline	0.67102	0.66892	0.66916

Bulgarian - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	pateam	0.58060	0.67740	0.61760	0.64620	0.87100
2	QUST	0.54840	0.57580	0.55880	0.56720	0.70970
3	nihao	0.48390	0.58060	0.52940	0.55380	0.74190
4	DEMON	0.41940	0.51610	0.47060	0.49230	0.74190
5	Dhananjaya	0.32260	0.18180	0.17650	0.17910	0.58060
6	Blabla	0.25810	0.40000	0.41180	0.40580	0.74190
7	LPA	0.22580	0.35710	0.44120	0.39470	0.90320
7	LTG	0.22580	0.12900	0.11760	0.12310	0.45160
7	lha	0.22580	0.36110	0.38240	0.37140	0.77420
8	kzeky	0.19350	0.21880	0.20590	0.21210	0.54840
9	gowithnlp	0.16130	0.27690	0.52940	0.36360	0.61290
10	Baseline	0.06450	0.06450	0.05880	0.06150	0.64520
11	Malak	0.00000	0.00000	0.00000	0.00000	0.09680

Bulgarian - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	pateam	0.65800	0.31600	0.54300	0.34600
2	UNEDTeam	0.47800	0.40600	0.38500	0.39100
3	Apfelkuchen	0.52600	0.42900	0.30000	0.41200
4	NotMyNarrative	0.21100	0.38400	0.19500	0.38300
5	iLostTheCode	0.22400	0.36700	0.18800	0.31400
6	Irapuarani	0.00000	0.00000	0.08600	0.28000
7	Baseline	0.03800	0.15500	0.01400	0.08300
8	IRNLP	0.21000	0.39100	0.00700	0.04200

Bulgarian - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	OmarElHerraoui	0.69936	0.71289	0.70572
2	amoriodi	0.68162	0.72907	0.70430
3	telorbulat	0.67677	0.67249	0.67409
4	Baseline	0.61946	0.65360	0.63589

Russian - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	pateam	0.63950	0.65120	0.62920	0.64000	0.90700
2	nihao	0.58140	0.59770	0.58430	0.59090	0.88370
3	QUST	0.55810	0.57950	0.57300	0.57630	0.89530
4	DEMON	0.53490	0.54650	0.52810	0.53710	0.89530
5	LTG	0.41860	0.15910	0.15730	0.15820	0.50000
6	Dhananjaya	0.39530	0.40700	0.39330	0.40000	0.69770
7	gowithnlp	0.37210	0.40000	0.38200	0.39080	0.70930
8	YNUzwt	0.30230	0.15120	0.14610	0.14860	0.48840
9	Baseline	0.05810	0.05810	0.05620	0.05710	0.33720

Russian - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	pateam	0.66900	0.27200	0.46300	0.30800
2	UNEDTeam	0.61100	0.37500	0.41100	0.36900
3	Apfelkuchen	0.51600	0.31900	0.24800	0.37000
4	iLostTheCode	0.42800	0.26000	0.23400	0.17300
5	IRNLP	0.44200	0.31300	0.13100	0.30600
6	NotMyNarrative	0.16700	0.33400	0.12900	0.33000
7	YNUzwt	0.03100	0.17400	0.03100	0.17400
8	Baseline	0.04100	0.13800	0.01300	0.07000
9	belta8	0.00000	0.00000	0.00000	0.00000

Russian - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	YNUzwt	0.66438	0.65943	0.66126
2	Baseline	0.62504	0.66336	0.64332

Note that the leaderboard is updated every 20 minutes, therefore your submission may not be listed right away.

Contact

twitter.com/_proppy_
Register to the google group of the task to ask questions and interact with other participants

SemEval 2025 shared task on "Multilingual Characterization and Extraction of Narratives from Online News"