Leaderboard

On December 6th 2024, a hotfix for subtask 1 has been applied to resolve an issue with order-sensitivity in the Exact Match Ratio. Predictions with the same fine-grained labels, regardless of order, are now treated equally.

English - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	nihao	0.50550	0.54950	0.50000	0.52360	0.91210
2	DEMON	0.46150	0.52750	0.48000	0.50260	0.91210
3	QUST	0.42860	0.47310	0.44000	0.45600	0.93410
4	kositeam	0.39560	0.42860	0.39000	0.40840	0.85710
5	pateam	0.38460	0.08600	0.08000	0.08290	0.68130
6	Blabla	0.34070	0.40220	0.37000	0.38540	0.82420
6	Mekky	0.34070	0.39360	0.37000	0.38140	0.80220
6	MikasaAckerman	0.34070	0.38460	0.35000	0.36650	0.78020
6	kzeky	0.34070	0.37360	0.34000	0.35600	0.84620
7	Agent667	0.32970	0.37250	0.38000	0.37620	0.86810
8	LTG	0.31870	0.07690	0.07000	0.07330	0.78020
8	auriza	0.31870	0.36110	0.39000	0.37500	0.79120
9	AtharvaK	0.30770	0.49330	0.37000	0.42290	0.83520
9	SZM	0.30770	0.43240	0.32000	0.36780	0.89010
9	ToTeamOrNotToTeam	0.30770	0.35160	0.32000	0.33510	0.75820
9	gowithnlp	0.30770	0.37360	0.34000	0.35600	0.84620
9	mbzuaijcbcruz	0.30770	0.13040	0.12000	0.12500	0.58240
10	DinDon	0.29670	0.37780	0.34000	0.35790	0.78020
11	Cendekiaaa	0.28570	0.34000	0.34000	0.34000	0.74730
11	NarrativeMiners	0.28570	0.28570	0.26000	0.27230	0.80220
11	alamo	0.28570	0.32320	0.32000	0.32160	0.90110
11	amoriodi	0.28570	0.13190	0.12000	0.12570	0.67030
12	Saeed	0.27470	0.30430	0.28000	0.29170	0.89010
12	m1nadzuki	0.27470	0.29670	0.27000	0.28270	0.89010
13	Malak	0.26370	0.26880	0.25000	0.25910	0.73630
13	Sarfraz307	0.26370	0.27470	0.25000	0.26180	0.70330
13	zizo	0.26370	0.30480	0.32000	0.31220	0.81320
14	ShehanMBZUAI	0.25270	0.29290	0.29000	0.29150	0.79120
15	news88readers	0.24180	0.29670	0.27000	0.28270	0.78020
16	lha	0.23080	0.22950	0.28000	0.25230	0.74730
16	testerbester	0.23080	0.23400	0.22000	0.22680	0.64840
17	Grimoires	0.21980	0.23160	0.22000	0.22560	0.73630
17	Tuebingen	0.21980	0.29290	0.29000	0.29150	0.75820
17	UtkarshAgarwal	0.21980	0.26370	0.24000	0.25130	0.80220
18	KevinMBZUAI	0.20880	0.27370	0.26000	0.26670	0.26370
19	AMUPD	0.19780	0.24720	0.22000	0.23280	0.83520
19	MeowgicalWitch	0.19780	0.20650	0.19000	0.19790	0.91210
19	Mon	0.19780	0.23810	0.30000	0.26550	0.72530
19	cdos	0.19780	0.24000	0.30000	0.26670	0.65930
20	BoiCoDon	0.17580	0.15450	0.17000	0.16190	0.74730
20	MARK	0.17580	0.19570	0.18000	0.18750	0.62640
20	mosalah	0.17580	0.20880	0.19000	0.19900	0.76920
21	Harum	0.16480	0.18480	0.17000	0.17710	0.64840
21	IvanRodkin	0.16480	0.18950	0.18000	0.18460	0.72530
21	LPA	0.16480	0.20950	0.22000	0.21460	0.78020
22	arham306	0.15380	0.18480	0.17000	0.17710	0.64840
22	auroracx	0.15380	0.18680	0.17000	0.17800	0.74730
23	Asqii619	0.14290	0.17580	0.16000	0.16750	0.68130
23	Survivors	0.14290	0.15050	0.14000	0.14510	0.80220
23	hocpocTUCN	0.14290	0.15380	0.14000	0.14660	0.80220
24	LTX	0.13190	0.14290	0.13000	0.13610	0.80220
24	nnn	0.13190	0.14290	0.13000	0.13610	0.80220
25	Baseline	0.12090	0.12090	0.11000	0.11520	0.80220
25	Dhananjaya	0.12090	0.12090	0.11000	0.11520	0.80220
25	Fane	0.12090	0.12090	0.11000	0.11520	0.80220
25	Kavya	0.12090	0.12090	0.11000	0.11520	0.80220
25	PAI	0.12090	0.12090	0.11000	0.11520	0.80220
25	YNUHPCC	0.12090	0.12090	0.11000	0.11520	0.80220
25	btf	0.12090	0.12090	0.11000	0.11520	0.80220
25	esbay	0.12090	0.12090	0.11000	0.11520	0.80220
25	noob	0.12090	0.12090	0.11000	0.11520	0.80220
26	4oclock	0.10990	0.26730	0.27000	0.26870	0.69230
27	adithjrajeev	0.08790	0.13190	0.12000	0.12570	0.80220
28	SavvyScout	0.05490	0.11670	0.30000	0.16810	0.82420
29	Gzxy	0.04400	0.04400	0.04000	0.04190	0.14290
30	CAISA	0.03300	0.03300	0.03000	0.03140	0.78020
31	Besher	0.00000	0.00000	0.00000	0.00000	0.00000
31	Bilal	0.00000	0.00000	0.00000	0.00000	0.00000
31	Genadium	0.00000	0.00000	0.00000	0.00000	0.00000
31	Irapuarani	0.00000	0.00000	0.00000	0.00000	0.00000
31	MME	0.00000	0.00000	0.00000	0.00000	0.00000
31	MyriamB	0.00000	0.00000	0.00000	0.00000	0.00000
31	NLPPraktikumWS2025	0.00000	0.00000	0.00000	0.00000	0.00000
32	NarrativeNexus	0.00000	1.00000	0.00000	0.00000	0.76920
33	RachelBakerPalen22	0.00000	0.00000	0.00000	0.00000	0.00000
33	iLostTheCode	0.00000	0.00000	0.00000	0.00000	0.00000
33	insun606	0.00000	0.00000	0.00000	0.00000	0.00000
33	telorbulat	0.00000	0.00000	0.00000	0.00000	0.00000

English - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	pateam	0.49200	0.31900	0.30400	0.29200
2	Abdulaziz	0.26800	0.44300	0.26800	0.44300
2	CodeExperts	0.26800	0.44300	0.26800	0.44300
2	Dhabi	0.19500	0.39600	0.26800	0.44300
2	GATENLP	0.26800	0.44300	0.26800	0.44300
2	Irapuarani	0.26800	0.44300	0.26800	0.44300
2	KostasThesis2025	0.26800	0.44300	0.26800	0.44300
2	NLPPraktikumWS2025	0.26800	0.44300	0.26800	0.44300
2	RachelBakerPalen22	0.19400	0.21300	0.26800	0.44300
2	iLostTheCode	0.26800	0.44300	0.26800	0.44300
2	m1nadzuki	0.26800	0.44300	0.26800	0.44300
3	NotMyNarrative	0.28200	0.42600	0.25900	0.42600
4	Alaa	0.22300	0.34800	0.18800	0.37700
5	Blabla	0.19600	0.35000	0.14000	0.33200
6	DUTtask10	0.14100	0.33200	0.09800	0.29700
6	NarrativeMiners	0.09800	0.29700	0.09800	0.29700
7	EGwal1A	0.12000	0.26700	0.08200	0.22700
8	CoSmOS	0.12800	0.24600	0.07900	0.18400
9	belta8	0.22600	0.27600	0.06700	0.18100
10	BaSingSe	0.10800	0.24200	0.05600	0.21800
11	PAI	0.03500	0.13300	0.01400	0.06600
12	Asqii619	0.00000	0.00000	0.00000	0.00000
13	Baseline	0.10600	0.26700	0.00000	0.00000
14	Malak	0.00000	0.00000	0.00000	0.00000
15	news88readers	0.10600	0.26700	0.00000	0.00000

English - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	DAVICHI	0.76044	0.73454	0.74699
2	Besher	0.75910	0.73481	0.74662
3	Edd	0.75922	0.73416	0.74607
4	MME	0.78395	0.71090	0.74533
5	Bilal	0.75648	0.73447	0.74517
6	Genadium	0.75398	0.73138	0.74233
7	NarrativeMiners	0.72860	0.74883	0.73848
8	notmena	0.75088	0.72337	0.73661
9	telorbulat	0.73944	0.72474	0.73181
10	OmarElHerraoui	0.73826	0.71351	0.72543
11	SholpanBolatzhanova	0.72479	0.72540	0.72494
12	insun606	0.72366	0.70875	0.71595
13	m1nadzuki	0.72482	0.70139	0.71260
14	GPLSICORTEX	0.71671	0.69973	0.70781
15	honey	0.71193	0.69861	0.70506
16	Kavya	0.76003	0.65646	0.70438
17	MyriamB	0.70125	0.70137	0.70114
18	amoriodi	0.68856	0.71412	0.70103
19	DUTtask10	0.70203	0.69695	0.69936
20	AJ805	0.65667	0.68130	0.66871
21	Baseline	0.65540	0.67957	0.66719
21	news88readers	0.65540	0.67957	0.66719
22	NarrativeNexus	0.00000	0.00000	0.00000
22	hacker	0.00000	0.00000	0.00000

Portuguese - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	QUST	0.72410	0.77590	0.72580	0.75000	0.90520
2	DEMON	0.70690	0.76720	0.71770	0.74170	0.96550
2	LTG	0.70690	0.26720	0.25000	0.25830	0.37930
3	Blabla	0.66380	0.77480	0.69350	0.73190	0.94830
3	nihao	0.66380	0.73280	0.68550	0.70830	0.88790
4	LPA	0.62930	0.67740	0.67740	0.67740	0.85340
4	lha	0.62930	0.67230	0.64520	0.65840	0.85340
5	Grimoires	0.59480	0.62810	0.61290	0.62040	0.83620
6	Dhananjaya	0.57760	0.40160	0.39520	0.39840	0.52590
7	Sarfraz307	0.56900	0.59320	0.56450	0.57850	0.68970
8	ToTeamOrNotToTeam	0.55170	0.57760	0.54030	0.55830	0.75000
9	Baseline	0.44830	0.48280	0.45160	0.46670	0.24140
10	gowithnlp	0.43970	0.48280	0.67740	0.56380	0.80170
11	amoriodi	0.42240	0.18100	0.16940	0.17500	0.26720
12	kzeky	0.27590	0.29660	0.28230	0.28930	0.46550
13	Malak	0.00000	0.00000	0.00000	0.00000	0.25860

Portuguese - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	NotMyNarrative	0.33200	0.31500	0.20500	0.24900
2	iLostTheCode	0.29500	0.42000	0.11400	0.26100
3	Irapuarani	0.00000	0.00000	0.02900	0.16700
3	KostasThesis2025	0.02900	0.16700	0.02900	0.16700
4	Baseline	0.06700	0.18800	0.01000	0.05600

Portuguese - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	telorbulat	0.75258	0.72481	0.73806
2	OmarElHerraoui	0.72418	0.73771	0.73058
3	amoriodi	0.71370	0.73489	0.72393
4	Baseline	0.65332	0.67644	0.66453

Hindi - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	DEMON	0.47500	0.54870	0.49350	0.51970	0.79290
2	Blabla	0.47140	0.56620	0.50000	0.53100	0.78930
3	LTG	0.41070	0.15000	0.13640	0.14290	0.38570
4	QUST	0.30360	0.38360	0.37990	0.38170	0.65360
5	LPA	0.29640	0.35530	0.36690	0.36100	0.63570
6	lha	0.28930	0.35080	0.37010	0.36020	0.58570
7	kzeky	0.23570	0.26690	0.26950	0.26820	0.55000
8	Dhananjaya	0.18210	0.09950	0.07140	0.08320	0.32860
9	Baseline	0.17140	0.18930	0.17210	0.18030	0.34290
10	gowithnlp	0.13210	0.23230	0.34090	0.27630	0.54290
11	Malak	0.01790	0.05290	0.02920	0.03770	0.37140

Hindi - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	iLostTheCode	0.23200	0.34600	0.13200	0.28500
2	NotMyNarrative	0.17400	0.18500	0.07900	0.08700
3	GATENLP	0.17100	0.27300	0.07700	0.17900
4	Baseline	0.10000	0.26200	0.05100	0.18700
5	Irapuarani	0.02900	0.16700	0.02900	0.16700

Hindi - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	amoriodi	0.72694	0.69082	0.70803
2	HitManZombie	0.68260	0.69997	0.69098
3	telorbulat	0.69762	0.67638	0.68616
4	OmarElHerraoui	0.69865	0.66637	0.68181
5	Baseline	0.67102	0.66892	0.66916

Bulgarian - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	QUST	0.54840	0.57580	0.55880	0.56720	0.70970
2	DEMON	0.38710	0.48390	0.44120	0.46150	0.77420
3	Blabla	0.25810	0.40000	0.41180	0.40580	0.74190
4	LPA	0.22580	0.35710	0.44120	0.39470	0.90320
4	lha	0.22580	0.36110	0.38240	0.37140	0.77420
5	LTG	0.19350	0.16130	0.14710	0.15380	0.51610
5	kzeky	0.19350	0.21880	0.20590	0.21210	0.54840
6	Dhananjaya	0.16130	0.27590	0.23530	0.25400	0.67740
6	gowithnlp	0.16130	0.27690	0.52940	0.36360	0.61290
7	Baseline	0.06450	0.06450	0.05880	0.06150	0.64520
8	Malak	0.00000	0.00000	0.00000	0.00000	0.09680

Bulgarian - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	NotMyNarrative	0.22500	0.29700	0.11800	0.26300
2	Irapuarani	0.00000	0.00000	0.08600	0.28000
3	Baseline	0.03800	0.15500	0.01400	0.08300

Bulgarian - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	OmarElHerraoui	0.69936	0.71289	0.70572
2	amoriodi	0.68162	0.72907	0.70430
3	telorbulat	0.67677	0.67249	0.67409
4	Baseline	0.61946	0.65360	0.63589

Russian - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	QUST	0.55810	0.57950	0.57300	0.57630	0.89530
2	DEMON	0.53490	0.54650	0.52810	0.53710	0.89530
3	LTG	0.34880	0.16280	0.15730	0.16000	0.53490
4	Baseline	0.05810	0.05810	0.05620	0.05710	0.33720

Russian - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 samples	F1 st. dev. samples
1	Baseline	0.04100	0.13800	0.01300	0.07000

Russian - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	Baseline	0.62504	0.66336	0.64332

Note that the leaderboard is updated every 20 minutes, therefore your submission may not be listed right away.

Contact

twitter.com/_proppy_
Register to the google group of the task to ask questions and interact with other participants

SemEval 2025 shared task on "Multilingual Characterization and Extraction of Narratives from Online News"