Leaderboard

On December 6th 2024, a hotfix for subtask 1 has been applied to resolve an issue with order-sensitivity in the Exact Match Ratio. Predictions with the same fine-grained labels, regardless of order, are now treated equally.

English - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	nihao	0.50550	0.54950	0.50000	0.52360	0.91210
2	DEMON	0.40660	0.47730	0.42000	0.44680	0.92310
3	kositeam	0.39560	0.42860	0.39000	0.40840	0.85710
4	QUST	0.38460	0.40660	0.37000	0.38740	0.80220
5	pateam	0.37360	0.07690	0.07000	0.07330	0.59340
6	Blabla	0.34070	0.40220	0.37000	0.38540	0.82420
6	Mekky	0.34070	0.39360	0.37000	0.38140	0.80220
6	MikasaAckerman	0.34070	0.38460	0.35000	0.36650	0.78020
6	kzeky	0.34070	0.37360	0.34000	0.35600	0.84620
7	Agent667	0.32970	0.37250	0.38000	0.37620	0.86810
8	auriza	0.31870	0.36110	0.39000	0.37500	0.79120
9	AtharvaK	0.30770	0.49330	0.37000	0.42290	0.83520
9	SZM	0.30770	0.43240	0.32000	0.36780	0.89010
9	ToTeamOrNotToTeam	0.30770	0.35160	0.32000	0.33510	0.75820
9	mbzuaijcbcruz	0.30770	0.13040	0.12000	0.12500	0.58240
10	DinDon	0.29670	0.37780	0.34000	0.35790	0.78020
11	Cendekiaaa	0.28570	0.34000	0.34000	0.34000	0.74730
11	NarrativeMiners	0.28570	0.28570	0.26000	0.27230	0.80220
11	alamo	0.28570	0.32320	0.32000	0.32160	0.90110
11	amoriodi	0.28570	0.13190	0.12000	0.12570	0.67030
12	Saeed	0.27470	0.30430	0.28000	0.29170	0.89010
12	m1nadzuki	0.27470	0.29670	0.27000	0.28270	0.89010
13	Malak	0.26370	0.26880	0.25000	0.25910	0.73630
13	Sarfraz307	0.26370	0.27470	0.25000	0.26180	0.70330
14	ShehanMBZUAI	0.25270	0.29290	0.29000	0.29150	0.79120
15	lha	0.23080	0.22950	0.28000	0.25230	0.74730
15	testerbester	0.23080	0.23400	0.22000	0.22680	0.64840
16	Grimoires	0.21980	0.23160	0.22000	0.22560	0.73630
16	Tuebingen	0.21980	0.29290	0.29000	0.29150	0.75820
16	UtkarshAgarwal	0.21980	0.26370	0.24000	0.25130	0.80220
16	news88readers	0.21980	0.27470	0.25000	0.26180	0.78020
17	KevinMBZUAI	0.20880	0.27370	0.26000	0.26670	0.26370
18	AMUPD	0.19780	0.24720	0.22000	0.23280	0.83520
18	MeowgicalWitch	0.19780	0.20650	0.19000	0.19790	0.91210
18	Mon	0.19780	0.23810	0.30000	0.26550	0.72530
18	cdos	0.19780	0.24000	0.30000	0.26670	0.65930
19	BoiCoDon	0.17580	0.15450	0.17000	0.16190	0.74730
19	MARK	0.17580	0.19570	0.18000	0.18750	0.62640
19	mosalah	0.17580	0.20880	0.19000	0.19900	0.76920
20	Harum	0.16480	0.18480	0.17000	0.17710	0.64840
20	IvanRodkin	0.16480	0.18950	0.18000	0.18460	0.72530
20	LPA	0.16480	0.20950	0.22000	0.21460	0.78020
21	arham306	0.15380	0.18480	0.17000	0.17710	0.64840
21	auroracx	0.15380	0.18680	0.17000	0.17800	0.74730
22	Asqii619	0.14290	0.17580	0.16000	0.16750	0.68130
22	Survivors	0.14290	0.15050	0.14000	0.14510	0.80220
23	LTX	0.13190	0.14290	0.13000	0.13610	0.80220
23	hocpocTUCN	0.13190	0.14290	0.13000	0.13610	0.82420
23	nnn	0.13190	0.14290	0.13000	0.13610	0.80220
24	Baseline	0.12090	0.12090	0.11000	0.11520	0.80220
24	Dhananjaya	0.12090	0.12090	0.11000	0.11520	0.80220
24	Kavya	0.12090	0.12090	0.11000	0.11520	0.80220
24	PAI	0.12090	0.12090	0.11000	0.11520	0.80220
24	btf	0.12090	0.12090	0.11000	0.11520	0.80220
24	esbay	0.12090	0.12090	0.11000	0.11520	0.80220
24	gowithnlp	0.12090	0.12090	0.11000	0.11520	0.80220
24	noob	0.12090	0.12090	0.11000	0.11520	0.80220
25	4oclock	0.10990	0.26730	0.27000	0.26870	0.69230
26	SavvyScout	0.05490	0.11670	0.30000	0.16810	0.82420
27	CAISA	0.03300	0.03300	0.03000	0.03140	0.78020
28	Besher	0.00000	0.00000	0.00000	0.00000	0.00000
28	Bilal	0.00000	0.00000	0.00000	0.00000	0.00000
28	Genadium	0.00000	0.00000	0.00000	0.00000	0.00000
28	Irapuarani	0.00000	0.00000	0.00000	0.00000	0.00000
28	MME	0.00000	0.00000	0.00000	0.00000	0.00000
28	MyriamB	0.00000	0.00000	0.00000	0.00000	0.00000
28	NLPPraktikumWS2025	0.00000	0.00000	0.00000	0.00000	0.00000
29	NarrativeNexus	0.00000	1.00000	0.00000	0.00000	0.76920
30	RachelBakerPalen22	0.00000	0.00000	0.00000	0.00000	0.00000
30	adithjrajeev	0.00000	0.00000	0.00000	0.00000	0.00000
30	iLostTheCode	0.00000	0.00000	0.00000	0.00000	0.00000
30	insun606	0.00000	0.00000	0.00000	0.00000	0.00000
30	telorbulat	0.00000	0.00000	0.00000	0.00000	0.00000

English - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 macro fine	F1 st. dev. fine
1	pateam	0.49200	0.31900	0.30400	0.29200
2	Abdulaziz	0.26800	0.44300	0.26800	0.44300
2	CodeExperts	0.26800	0.44300	0.26800	0.44300
2	GATENLP	0.26800	0.44300	0.26800	0.44300
2	Irapuarani	0.26800	0.44300	0.26800	0.44300
2	NLPPraktikumWS2025	0.26800	0.44300	0.26800	0.44300
2	RachelBakerPalen22	0.19400	0.21300	0.26800	0.44300
2	iLostTheCode	0.26800	0.44300	0.26800	0.44300
3	NotMyNarrative	0.28200	0.42600	0.25900	0.42600
4	Alaa	0.22300	0.34800	0.18800	0.37700
5	Blabla	0.19600	0.35000	0.14000	0.33200
6	DUTtask10	0.14100	0.33200	0.09800	0.29700
6	NarrativeMiners	0.09800	0.29700	0.09800	0.29700
7	CoSmOS	0.12800	0.24600	0.07900	0.18400
8	belta8	0.22600	0.27600	0.06700	0.18100
9	EGwal1A	0.12900	0.20400	0.06100	0.09500
10	BaSingSe	0.10800	0.24200	0.05600	0.21800
11	PAI	0.03500	0.13300	0.01400	0.06600
12	Asqii619	0.00000	0.00000	0.00000	0.00000
13	Baseline	0.10600	0.26700	0.00000	0.00000
14	Dhabi	0.00000	0.00000	0.00000	0.00000
14	Malak	0.00000	0.00000	0.00000	0.00000
15	news88readers	0.10600	0.26700	0.00000	0.00000

English - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	DAVICHI	0.76044	0.73454	0.74699
2	Besher	0.75910	0.73481	0.74662
3	Edd	0.75922	0.73416	0.74607
4	MME	0.78395	0.71090	0.74533
5	Bilal	0.75648	0.73447	0.74517
6	Genadium	0.75398	0.73138	0.74233
7	NarrativeMiners	0.72860	0.74883	0.73848
8	notmena	0.75088	0.72337	0.73661
9	telorbulat	0.73944	0.72474	0.73181
10	OmarElHerraoui	0.73826	0.71351	0.72543
11	SholpanBolatzhanova	0.72479	0.72540	0.72494
12	insun606	0.72366	0.70875	0.71595
13	honey	0.71193	0.69861	0.70506
14	Kavya	0.76003	0.65646	0.70438
15	MyriamB	0.70125	0.70137	0.70114
16	amoriodi	0.68856	0.71412	0.70103
17	DUTtask10	0.70203	0.69695	0.69936
18	AJ805	0.65667	0.68130	0.66871
19	Baseline	0.65540	0.67957	0.66719
19	news88readers	0.65540	0.67957	0.66719
20	NarrativeNexus	0.00000	0.00000	0.00000
20	hacker	0.00000	0.00000	0.00000

Portuguese - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	Blabla	0.66380	0.77480	0.69350	0.73190	0.94830
1	nihao	0.66380	0.73280	0.68550	0.70830	0.88790
2	DEMON	0.65520	0.69830	0.65320	0.67500	0.94830
3	LPA	0.62930	0.67740	0.67740	0.67740	0.85340
3	lha	0.62930	0.67230	0.64520	0.65840	0.85340
4	Dhananjaya	0.57760	0.40160	0.39520	0.39840	0.52590
5	Sarfraz307	0.56900	0.59320	0.56450	0.57850	0.68970
6	ToTeamOrNotToTeam	0.55170	0.57760	0.54030	0.55830	0.75000
7	Baseline	0.44830	0.48280	0.45160	0.46670	0.24140
8	amoriodi	0.42240	0.18100	0.16940	0.17500	0.26720
9	gowithnlp	0.37070	0.45600	0.70970	0.55520	0.82760
10	kzeky	0.27590	0.29660	0.28230	0.28930	0.46550
11	Malak	0.00000	0.00000	0.00000	0.00000	0.25860

Portuguese - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 macro fine	F1 st. dev. fine
1	NotMyNarrative	0.33200	0.31500	0.20500	0.24900
2	iLostTheCode	0.29500	0.42000	0.11400	0.26100
3	Irapuarani	0.00000	0.00000	0.02900	0.16700
4	Baseline	0.06700	0.18800	0.01000	0.05600

Portuguese - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	telorbulat	0.75258	0.72481	0.73806
2	OmarElHerraoui	0.72418	0.73771	0.73058
3	amoriodi	0.71370	0.73489	0.72393
4	Baseline	0.65332	0.67644	0.66453

Hindi - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	Blabla	0.47140	0.56620	0.50000	0.53100	0.78930
2	DEMON	0.35710	0.41790	0.37990	0.39800	0.68930
3	LPA	0.29640	0.35530	0.36690	0.36100	0.63570
4	lha	0.28930	0.35080	0.37010	0.36020	0.58570
5	kzeky	0.23570	0.26690	0.26950	0.26820	0.55000
6	Dhananjaya	0.18210	0.09950	0.07140	0.08320	0.32860
7	Baseline	0.17140	0.18930	0.17210	0.18030	0.34290
8	gowithnlp	0.13210	0.23230	0.34090	0.27630	0.54290
9	Malak	0.01790	0.05290	0.02920	0.03770	0.37140

Hindi - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 macro fine	F1 st. dev. fine
1	iLostTheCode	0.23200	0.34600	0.13200	0.28500
2	GATENLP	0.17100	0.27300	0.07700	0.17900
3	NotMyNarrative	0.14800	0.19700	0.06900	0.11300
4	Baseline	0.10000	0.26200	0.05100	0.18700
5	Irapuarani	0.02900	0.16700	0.02900	0.16700

Hindi - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	amoriodi	0.72694	0.69082	0.70803
2	telorbulat	0.69762	0.67638	0.68616
3	OmarElHerraoui	0.69865	0.66637	0.68181
4	HitManZombie	0.67323	0.68954	0.68102
5	Baseline	0.67102	0.66892	0.66916

Bulgarian - Subtask 1

Rank	Team	Exact Match Ratio	micro P	micro R	micro F1	Accuracy for main role
1	DEMON	0.32260	0.37930	0.32350	0.34920	0.80650
2	Blabla	0.25810	0.40000	0.41180	0.40580	0.74190
3	LPA	0.22580	0.35710	0.44120	0.39470	0.90320
3	lha	0.22580	0.36110	0.38240	0.37140	0.77420
4	kzeky	0.19350	0.21880	0.20590	0.21210	0.54840
5	Dhananjaya	0.16130	0.27590	0.23530	0.25400	0.67740
5	gowithnlp	0.16130	0.27690	0.52940	0.36360	0.61290
6	Baseline	0.06450	0.06450	0.05880	0.06150	0.64520
7	Malak	0.00000	0.00000	0.00000	0.00000	0.09680

Bulgarian - Subtask 2

Rank	Team	F1 macro coarse	F1 st. dev. coarse	F1 macro fine	F1 st. dev. fine
1	NotMyNarrative	0.18600	0.27800	0.11700	0.25100
2	Irapuarani	0.00000	0.00000	0.08600	0.28000
3	Baseline	0.03800	0.15500	0.01400	0.08300

Bulgarian - Subtask 3

Rank	Team	Precision	Recall	F1 macro
1	OmarElHerraoui	0.69936	0.71289	0.70572
2	amoriodi	0.68162	0.72907	0.70430
3	telorbulat	0.67677	0.67249	0.67409
4	Baseline	0.61946	0.65360	0.63589

Note that the leaderboard is updated every 20 minutes, therefore your submission may not be listed right away.

Contact

twitter.com/_proppy_
Register to the google group of the task to ask questions and interact with other participants

SemEval 2025 shared task on "Multilingual Characterization and Extraction of Narratives from Online News"