Leaderboard

Official SemEval 2024 task 4 Test Leaderboards

SemEval 2024 task 4 Dev Leaderboards (until Feb. 1 2024)

Post SemEval Leaderboards on the Test set (Work in Progress)

English - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	914isthebest	0.75247	0.68419	0.83590
2	BCAmirs	0.69857	0.66786	0.73223
3	OtterlyObsessedWithSemantics	0.69738	0.64801	0.75490
4	TUMnlp	0.67384	0.63781	0.71419
5	GreyBox	0.66998	0.65248	0.68844
6	Phoenix	0.66391	0.70398	0.62815
7	NLPNCHU	0.66271	0.60990	0.72552
8	Puer	0.66026	0.64833	0.67264
9	EURECOM	0.65702	0.65989	0.65417
10	HierarchyEverywhere	0.65286	0.63041	0.67697
11	SuteAlbastre	0.65234	0.63262	0.67334
12	ShefCDTeam	0.65088	0.70082	0.60758
13	UMUTeam	0.64774	0.70817	0.59681
14	RDproj	0.64288	0.57530	0.72846
15	I2CUHU	0.64162	0.57088	0.73237
16	nowhash	0.64096	0.61167	0.67320
17	whatdoyoumeme	0.63579	0.58301	0.69908
18	Pauk	0.62660	0.71572	0.55722
19	MAD2024	0.62541	0.58416	0.67292
20	IUSTNLPLAB	0.62479	0.63194	0.61780
21	CLaC	0.62161	0.57563	0.67557
22	IITK	0.61722	0.54040	0.71950
23	LomonosovMSU	0.61339	0.71154	0.53903
24	SoftMiner	0.60656	0.64947	0.56897
25	MagnumJUCSE	0.60341	0.54655	0.67348
26	CVCoders	0.59727	0.52563	0.69152
27	27	0.59433	0.61711	0.57317
28	BAMBAS	0.58400	0.75946	0.47440
29	Elyas	0.57798	0.66227	0.51273
30	MemeSifters	0.57454	0.57619	0.57289
31	fralak	0.56521	0.48960	0.66844
32	IIITG	0.52569	0.61429	0.45943
33	Two	0.52197	0.52639	0.51763
34	Scalar	0.50481	0.43277	0.60562
35	SINAI	0.42519	0.31212	0.66676
36	McRock	0.42275	0.30141	0.70761
37	Baseline	0.36865	0.47711	0.30036
38	WhatsaMeme	0.34663	0.34728	0.34597
39	IIMAS1UTM1LaSalle	0.19917	0.75506	0.11472
40	THARA	0.00933	0.24286	0.00476

English - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	HierarchyEverywhere	0.74592	0.86682	0.65461
2	NLPNCHU	0.70677	0.78164	0.64498
3	BCAmirs	0.70497	0.78374	0.64059
4	Phoenix	0.69666	0.73742	0.66018
5	SuteAlbastre	0.69508	0.67193	0.71988
6	UMUTeam	0.69003	0.76763	0.62669
7	iLostTheCode	0.68425	0.64825	0.72448
8	TUMnlp	0.67717	0.78066	0.59790
9	Pauk	0.67529	0.74503	0.61748
10	IITK	0.67143	0.66954	0.67334
11	CodeMeme	0.66618	0.60655	0.73882
12	LomonosovMSU	0.65610	0.79153	0.56024
13	CLaC	0.64377	0.64895	0.63867
14	BERTastic	0.61338	0.81581	0.49144
15	Elyas	0.61158	0.62982	0.59437
16	27	0.54232	0.62451	0.47924
17	BDA	0.50388	0.51485	0.49337
18	Baseline	0.44706	0.68778	0.33116
18	CVCoders	0.44706	0.68778	0.33116
19	WhatsaMeme	0.36592	0.31342	0.43955

English - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	LMEME	0.83637	0.86000
2	iLostTheCode	0.81884	0.84167
3	DUTIR938	0.81462	0.84000
4	SuteAlbastre	0.80964	0.83500
5	BCAmirs	0.80337	0.82500
6	EverythingWarGia	0.80082	0.82833
7	Snarci	0.79860	0.82667
8	BDA	0.79290	0.82333
9	NLPNCHU	0.78803	0.82167
10	UMUTeam	0.78660	0.80667
11	TUMnlp	0.78413	0.80167
12	CodeMeme	0.78195	0.80667
13	LomonosovMSU	0.77226	0.79833
14	BERTastic	0.71582	0.76167
15	Hidetsune	0.71353	0.79000
16	Scalar	0.70151	0.75333
17	CVCoders	0.67544	0.70833
18	SheffieldVeraAI	0.64197	0.68667
19	HierarchyEverywhere	0.56309	0.66167
20	WhatsaMeme	0.51486	0.53000
21	nowhash	0.49845	0.51500
22	IITK	0.48336	0.49000
23	Baseline	0.25000	0.33333

Bulgarian - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	OtterlyObsessedWithSemantics	0.56833	0.51955	0.62722
2	RDproj	0.54133	0.43490	0.71674
3	NLPNCHU	0.51744	0.53578	0.50031
4	MagnumJUCSE	0.49986	0.47027	0.53342
5	nowhash	0.48627	0.46007	0.51563
6	Phoenix	0.48411	0.50809	0.46229
7	IITK	0.48353	0.39586	0.62109
8	MemeSifters	0.48125	0.49076	0.47210
9	GreyBox	0.47635	0.43821	0.52177
10	whatdoyoumeme	0.47257	0.50207	0.44635
11	HierarchyEverywhere	0.46757	0.48301	0.45310
12	fralak	0.46411	0.37360	0.61251
13	914isthebest	0.46309	0.47693	0.45003
14	I2CUHU	0.45565	0.37808	0.57327
15	SuteAlbastre	0.45318	0.43729	0.47026
16	CLaC	0.44917	0.40010	0.51196
17	BCAmirs	0.44834	0.38667	0.53342
18	ShefCDTeam	0.40283	0.44786	0.36603
19	27	0.39262	0.43365	0.35868
20	CVCoders	0.38433	0.38672	0.38197
21	EURECOM	0.34483	0.36729	0.32495
22	SINAI	0.34118	0.21351	0.84856
23	Baseline	0.28377	0.31881	0.25567
24	IIMAS1UTM1LaSalle	0.18345	0.65414	0.10668

Bulgarian - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	Phoenix	0.65638	0.67555	0.63828
2	IITK	0.64879	0.66414	0.63414
3	BCAmirs	0.62693	0.70278	0.56586
4	CLaC	0.62528	0.63427	0.61655
5	SuteAlbastre	0.61074	0.65960	0.56862
6	27	0.56332	0.71315	0.46552
7	NLPNCHU	0.54864	0.70691	0.44828
8	BERTastic	0.54358	0.81164	0.40862
9	Baseline	0.50000	0.80428	0.36276
10	BDA	0.48341	0.52265	0.44966
11	HierarchyEverywhere	0.46414	0.67080	0.35483

Bulgarian - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	LMEME	0.67100	0.81000
2	Snarci	0.66777	0.84000
3	BERTastic	0.66212	0.75000
4	BCAmirs	0.64719	0.77000
5	NLPNCHU	0.64706	0.82000
6	MemeSifters	0.61143	0.83000
7	SuteAlbastre	0.59448	0.65000
8	BDA	0.57143	0.64000
9	SheffieldVeraAI	0.53619	0.57000
10	CVCoders	0.51637	0.74000
11	HierarchyEverywhere	0.48547	0.63000
12	IITK	0.47256	0.53000
13	DUTIR938	0.43414	0.57000
14	nowhash	0.43363	0.45000
15	Hidetsune	0.32670	0.33000
16	Baseline	0.16667	0.20000

North Macedonian - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	OtterlyObsessedWithSemantics	0.51244	0.51824	0.50677
2	RDproj	0.49894	0.43814	0.57934
3	MagnumJUCSE	0.48267	0.48568	0.47970
4	Phoenix	0.46615	0.62218	0.37269
5	fralak	0.46441	0.35882	0.65806
6	NLPNCHU	0.46165	0.54622	0.39975
7	I2CUHU	0.44230	0.39238	0.50677
8	EURECOM	0.44161	0.52000	0.38376
9	MemeSifters	0.44073	0.53915	0.37269
10	GreyBox	0.43435	0.43955	0.42927
11	nowhash	0.42558	0.41395	0.43788
12	HierarchyEverywhere	0.41713	0.48609	0.36531
13	IITK	0.41217	0.32662	0.55843
14	CLaC	0.39471	0.37081	0.42189
15	BCAmirs	0.39298	0.33164	0.48216
16	27	0.37085	0.44852	0.31611
17	914isthebest	0.36860	0.40145	0.34071
18	whatdoyoumeme	0.36156	0.39852	0.33087
19	ShefCDTeam	0.35451	0.44362	0.29520
20	CVCoders	0.31222	0.28901	0.33948
21	Baseline	0.30692	0.31403	0.30012
22	SINAI	0.30109	0.18312	0.84625
23	SuteAlbastre	0.27662	0.16785	0.78598
24	IIMAS1UTM1LaSalle	0.13704	0.52893	0.07872

North Macedonian - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	Phoenix	0.69844	0.76676	0.64130
2	CLaC	0.68930	0.69923	0.67963
3	IITK	0.66667	0.72465	0.61728
4	BCAmirs	0.63681	0.75019	0.55320
5	SuteAlbastre	0.62496	0.68130	0.57723
6	27	0.62421	0.69522	0.56636
7	BERTastic	0.57329	0.86590	0.42849
8	Baseline	0.55525	0.90219	0.40103
9	BDA	0.50139	0.54619	0.46339
10	NLPNCHU	0.48707	0.70575	0.37185
11	HierarchyEverywhere	0.35693	0.68903	0.24085

North Macedonian - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	BERTastic	0.68627	0.84000
2	MemeSifters	0.66033	0.90000
3	LMEME	0.59077	0.78000
4	BDA	0.57765	0.74000
5	CVCoders	0.57653	0.79000
6	BCAmirs	0.56098	0.77000
7	NLPNCHU	0.52000	0.79000
8	HierarchyEverywhere	0.50624	0.62000
9	IITK	0.48547	0.63000
10	Snarci	0.47917	0.72000
11	DUTIR938	0.46875	0.66000
12	SheffieldVeraAI	0.45790	0.51000
13	nowhash	0.42857	0.52000
14	Hidetsune	0.38942	0.46000
15	SuteAlbastre	0.17704	0.18000
16	Baseline	0.09091	0.10000

Arabic - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	NLPNCHU	0.47500	0.42817	0.53333
2	Phoenix	0.44479	0.39510	0.50877
3	IITK	0.42637	0.35514	0.53333
4	whatdoyoumeme	0.42379	0.32759	0.60000
5	fralak	0.42117	0.30421	0.68421
6	RDproj	0.41454	0.33624	0.54035
7	BCAmirs	0.39625	0.32035	0.51930
8	I2CUHU	0.39152	0.30368	0.55088
9	CLaC	0.38070	0.30803	0.49825
10	Baseline	0.35897	0.35000	0.36842
11	CVCoders	0.34115	0.27122	0.45965
12	27	0.33982	0.34286	0.33684
13	914isthebest	0.00000	0.00000	0.00000
13	EURECOM	0.00000	0.00000	0.00000
13	HierarchyEverywhere	0.00000	0.00000	0.00000
13	MagnumJUCSE	0.00000	0.00000	0.00000
13	MemeSifters	0.00000	0.00000	0.00000
13	OtterlyObsessedWithSemantics	0.00000	0.00000	0.00000
13	SINAI	0.00000	0.00000	0.00000
13	SuteAlbastre	0.00000	0.00000	0.00000
13	nowhash	0.00000	0.00000	0.00000

Arabic - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	IITK	0.53636	0.59237	0.49003
2	Phoenix	0.53378	0.53601	0.53156
3	BCAmirs	0.52613	0.55311	0.50166
4	CLaC	0.51401	0.49614	0.53322
5	27	0.49915	0.51408	0.48505
6	Baseline	0.48649	0.65000	0.38870
7	NLPNCHU	0.48323	0.59466	0.40698
8	BDA	0.41635	0.38248	0.45681
9	BERTastic	0.00000	0.00000	0.00000
9	HierarchyEverywhere	0.00000	0.00000	0.00000
9	SuteAlbastre	0.00000	0.00000	0.00000

Arabic - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	BDA	0.62137	0.65000
2	BCAmirs	0.61487	0.63125
3	SheffieldVeraAI	0.61031	0.61250
4	BERTastic	0.60276	0.60625
5	NLPNCHU	0.58518	0.59375
6	HierarchyEverywhere	0.56196	0.66875
7	MemeSifters	0.55654	0.69375
8	Snarci	0.55540	0.55625
9	Hidetsune	0.52825	0.54375
10	SuteAlbastre	0.50066	0.54375
11	nowhash	0.49831	0.53125
12	DUTIR938	0.46894	0.51875
13	IITK	0.46706	0.46875
14	LMEME	0.40940	0.42500
15	Baseline	0.22705	0.29375
15	CVCoders	0.22705	0.29375

Note that the leaderboard is updated every 20 minutes, therefore your submission may not be listed right away.

Contact

twitter.com/_proppy_
Register to the google group of the task to ask questions and interact with other participants

SemEval 2024 shared task on "Multilingual Detection of Persuasion Techniques in Memes"

Leaderboard

Official SemEval 2024 task 4 Test Leaderboards

SemEval 2024 task 4 Dev Leaderboards (until Feb. 1 2024)

Post SemEval Leaderboards on the Test set (Work in Progress)

English - Subtask 1

English - Subtask 2a

English - Subtask 2b

Bulgarian - Subtask 1

Bulgarian - Subtask 2a

Bulgarian - Subtask 2b

North Macedonian - Subtask 1

North Macedonian - Subtask 2a

North Macedonian - Subtask 2b

Arabic - Subtask 1

Arabic - Subtask 2a

Arabic - Subtask 2b

Contact