Leaderboard

Official SemEval 2024 task 4 Test Leaderboards

SemEval 2024 task 4 Dev Leaderboards (until Feb. 1 2024)

Post SemEval Leaderboards on the Test set (Work in Progress)

English - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	914isthebest	0.75247	0.68419	0.83590
2	Akerke	0.70382	0.70186	0.70579
3	BCAmirs	0.69857	0.66786	0.73223
4	OtterlyObsessedWithSemantics	0.69738	0.64801	0.75490
5	TUMnlp	0.67384	0.63781	0.71419
6	GreyBox	0.66998	0.65248	0.68844
7	SuteAlbastre	0.66675	0.61619	0.72636
8	Phoenix	0.66391	0.70398	0.62815
9	NLPNCHU	0.66271	0.60990	0.72552
10	Puer	0.66026	0.64833	0.67264
11	EURECOM	0.65702	0.65989	0.65417
12	HierarchyEverywhere	0.65286	0.63041	0.67697
13	ShefCDTeam	0.65088	0.70082	0.60758
14	UMUTeam	0.64774	0.70817	0.59681
15	RDproj	0.64288	0.57530	0.72846
16	I2CUHU	0.64162	0.57088	0.73237
17	nowhash	0.64096	0.61167	0.67320
18	whatdoyoumeme	0.63579	0.58301	0.69908
19	Pauk	0.62660	0.71572	0.55722
20	MAD2024	0.62541	0.58416	0.67292
21	IUSTNLPLAB	0.62479	0.63194	0.61780
22	CLaC	0.62161	0.57563	0.67557
23	IITK	0.61722	0.54040	0.71950
24	LomonosovMSU	0.61339	0.71154	0.53903
25	SoftMiner	0.60656	0.64947	0.56897
26	MagnumJUCSE	0.60341	0.54655	0.67348
27	CVCoders	0.59727	0.52563	0.69152
28	27	0.59433	0.61711	0.57317
29	BAMBAS	0.58400	0.75946	0.47440
30	Elyas	0.57798	0.66227	0.51273
31	MemeSifters	0.57454	0.57619	0.57289
32	fralak	0.56521	0.48960	0.66844
33	IIITG	0.52569	0.61429	0.45943
34	Two	0.52197	0.52639	0.51763
35	Scalar	0.50481	0.43277	0.60562
36	JM	0.48689	0.46433	0.51175
37	SINAI	0.42519	0.31212	0.66676
38	McRock	0.42275	0.30141	0.70761
39	Baseline	0.36865	0.47711	0.30036
40	WhatsaMeme	0.34663	0.34728	0.34597
41	IIMAS1UTM1LaSalle	0.19917	0.75506	0.11472
42	THARA	0.00933	0.24286	0.00476

English - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	HierarchyEverywhere	0.74592	0.86682	0.65461
2	SuteAlbastre	0.70830	0.69224	0.72512
3	NLPNCHU	0.70677	0.78164	0.64498
4	BCAmirs	0.70497	0.78374	0.64059
5	Phoenix	0.69666	0.73742	0.66018
6	UMUTeam	0.69003	0.76763	0.62669
7	iLostTheCode	0.68425	0.64825	0.72448
8	111	0.68157	0.68811	0.67516
9	TUMnlp	0.67717	0.78066	0.59790
10	Pauk	0.67529	0.74503	0.61748
11	IITK	0.67143	0.66954	0.67334
12	CodeMeme	0.66618	0.60655	0.73882
13	Elyas	0.65690	0.64038	0.67430
14	LomonosovMSU	0.65610	0.79153	0.56024
15	CLaC	0.64377	0.64895	0.63867
16	BERTastic	0.61338	0.81581	0.49144
17	27	0.54232	0.62451	0.47924
18	BDA	0.50388	0.51485	0.49337
19	JM	0.44784	0.46383	0.43291
20	Baseline	0.44706	0.68778	0.33116
20	CVCoders	0.44706	0.68778	0.33116
21	WhatsaMeme	0.36592	0.31342	0.43955

English - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	LMEME	0.83637	0.86000
2	iLostTheCode	0.81884	0.84167
3	DUTIR938	0.81462	0.84000
4	SuteAlbastre	0.80964	0.83500
5	BCAmirs	0.80337	0.82500
6	EverythingWarGia	0.80082	0.82833
7	Snarci	0.79860	0.82667
8	BDA	0.79290	0.82333
9	NLPNCHU	0.78803	0.82167
10	UMUTeam	0.78660	0.80667
11	TUMnlp	0.78413	0.80167
12	CodeMeme	0.78195	0.80667
13	LomonosovMSU	0.77226	0.79833
14	BERTastic	0.71582	0.76167
15	Hidetsune	0.71353	0.79000
16	Scalar	0.70151	0.75333
17	CVCoders	0.67544	0.70833
18	SheffieldVeraAI	0.64197	0.68667
19	JM	0.56570	0.67500
20	HierarchyEverywhere	0.56309	0.66167
21	WhatsaMeme	0.51486	0.53000
22	nowhash	0.49845	0.51500
23	IITK	0.48336	0.49000
24	Baseline	0.25000	0.33333

Bulgarian - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	OtterlyObsessedWithSemantics	0.56833	0.51955	0.62722
2	RDproj	0.54133	0.43490	0.71674
3	NLPNCHU	0.51744	0.53578	0.50031
4	MagnumJUCSE	0.49986	0.47027	0.53342
5	nowhash	0.48627	0.46007	0.51563
6	Phoenix	0.48411	0.50809	0.46229
7	IITK	0.48353	0.39586	0.62109
8	MemeSifters	0.48125	0.49076	0.47210
9	GreyBox	0.47635	0.43821	0.52177
10	whatdoyoumeme	0.47257	0.50207	0.44635
11	HierarchyEverywhere	0.46757	0.48301	0.45310
12	fralak	0.46411	0.37360	0.61251
13	914isthebest	0.46309	0.47693	0.45003
14	I2CUHU	0.45565	0.37808	0.57327
15	SuteAlbastre	0.45318	0.43729	0.47026
16	CLaC	0.44917	0.40010	0.51196
17	BCAmirs	0.44834	0.38667	0.53342
18	JM	0.43832	0.42832	0.44880
19	ShefCDTeam	0.40283	0.44786	0.36603
20	27	0.39262	0.43365	0.35868
21	CVCoders	0.38433	0.38672	0.38197
22	EURECOM	0.34483	0.36729	0.32495
23	SINAI	0.34118	0.21351	0.84856
24	Baseline	0.28377	0.31881	0.25567
25	IIMAS1UTM1LaSalle	0.18345	0.65414	0.10668

Bulgarian - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	Phoenix	0.65638	0.67555	0.63828
2	IITK	0.64879	0.66414	0.63414
3	BCAmirs	0.62693	0.70278	0.56586
4	CLaC	0.62528	0.63427	0.61655
5	SuteAlbastre	0.61074	0.65960	0.56862
6	27	0.56332	0.71315	0.46552
7	NLPNCHU	0.54864	0.70691	0.44828
8	BERTastic	0.54358	0.81164	0.40862
9	Baseline	0.50000	0.80428	0.36276
10	BDA	0.48341	0.52265	0.44966
11	HierarchyEverywhere	0.46414	0.67080	0.35483
12	JM	0.40569	0.45850	0.36379

Bulgarian - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	LMEME	0.67100	0.81000
2	Snarci	0.66777	0.84000
3	BERTastic	0.66212	0.75000
4	JM	0.65719	0.83000
5	BCAmirs	0.64719	0.77000
6	NLPNCHU	0.64706	0.82000
7	MemeSifters	0.61143	0.83000
8	SuteAlbastre	0.59448	0.65000
9	BDA	0.57143	0.64000
10	SheffieldVeraAI	0.53619	0.57000
11	CVCoders	0.51637	0.74000
12	HierarchyEverywhere	0.48547	0.63000
13	IITK	0.47256	0.53000
14	DUTIR938	0.43414	0.57000
15	nowhash	0.43363	0.45000
16	Hidetsune	0.32670	0.33000
17	Baseline	0.16667	0.20000

North Macedonian - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	OtterlyObsessedWithSemantics	0.51244	0.51824	0.50677
2	RDproj	0.49894	0.43814	0.57934
3	MagnumJUCSE	0.48267	0.48568	0.47970
4	Phoenix	0.46615	0.62218	0.37269
5	fralak	0.46441	0.35882	0.65806
6	NLPNCHU	0.46165	0.54622	0.39975
7	I2CUHU	0.44230	0.39238	0.50677
8	EURECOM	0.44161	0.52000	0.38376
9	MemeSifters	0.44073	0.53915	0.37269
10	GreyBox	0.43435	0.43955	0.42927
11	nowhash	0.42558	0.41395	0.43788
12	HierarchyEverywhere	0.41713	0.48609	0.36531
13	IITK	0.41217	0.32662	0.55843
14	CLaC	0.39471	0.37081	0.42189
15	BCAmirs	0.39298	0.33164	0.48216
16	JM	0.38397	0.40463	0.36531
17	27	0.37085	0.44852	0.31611
18	914isthebest	0.36860	0.40145	0.34071
19	whatdoyoumeme	0.36156	0.39852	0.33087
20	ShefCDTeam	0.35451	0.44362	0.29520
21	CVCoders	0.31222	0.28901	0.33948
22	Baseline	0.30692	0.31403	0.30012
23	SINAI	0.30109	0.18312	0.84625
24	SuteAlbastre	0.27662	0.16785	0.78598
25	IIMAS1UTM1LaSalle	0.13704	0.52893	0.07872

North Macedonian - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	Phoenix	0.69844	0.76676	0.64130
2	CLaC	0.68930	0.69923	0.67963
3	IITK	0.66667	0.72465	0.61728
4	BCAmirs	0.63681	0.75019	0.55320
5	SuteAlbastre	0.62496	0.68130	0.57723
6	27	0.62421	0.69522	0.56636
7	BERTastic	0.57329	0.86590	0.42849
8	Baseline	0.55525	0.90219	0.40103
9	BDA	0.50139	0.54619	0.46339
10	NLPNCHU	0.48707	0.70575	0.37185
11	JM	0.40903	0.48734	0.35240
12	HierarchyEverywhere	0.35693	0.68903	0.24085

North Macedonian - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	BERTastic	0.68627	0.84000
2	JM	0.66667	0.88000
3	MemeSifters	0.66033	0.90000
4	LMEME	0.59077	0.78000
5	BDA	0.57765	0.74000
6	CVCoders	0.57653	0.79000
7	BCAmirs	0.56098	0.77000
8	NLPNCHU	0.52000	0.79000
9	HierarchyEverywhere	0.50624	0.62000
10	IITK	0.48547	0.63000
11	Snarci	0.47917	0.72000
12	DUTIR938	0.46875	0.66000
13	SheffieldVeraAI	0.45790	0.51000
14	nowhash	0.42857	0.52000
15	Hidetsune	0.38942	0.46000
16	SuteAlbastre	0.17704	0.18000
17	Baseline	0.09091	0.10000

Arabic - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	NLPNCHU	0.47500	0.42817	0.53333
2	Phoenix	0.44479	0.39510	0.50877
3	SuteAlbastre	0.43323	0.37532	0.51228
4	IITK	0.42637	0.35514	0.53333
5	whatdoyoumeme	0.42379	0.32759	0.60000
6	fralak	0.42117	0.30421	0.68421
7	RDproj	0.41454	0.33624	0.54035
8	BCAmirs	0.39625	0.32035	0.51930
9	JM	0.39172	0.35860	0.43158
10	I2CUHU	0.39152	0.30368	0.55088
11	CLaC	0.38070	0.30803	0.49825
12	Baseline	0.35897	0.35000	0.36842
13	CVCoders	0.34115	0.27122	0.45965
14	27	0.33982	0.34286	0.33684
15	914isthebest	0.00000	0.00000	0.00000
15	EURECOM	0.00000	0.00000	0.00000
15	HierarchyEverywhere	0.00000	0.00000	0.00000
15	MagnumJUCSE	0.00000	0.00000	0.00000
15	MemeSifters	0.00000	0.00000	0.00000
15	OtterlyObsessedWithSemantics	0.00000	0.00000	0.00000
15	SINAI	0.00000	0.00000	0.00000
15	nowhash	0.00000	0.00000	0.00000

Arabic - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	IITK	0.53636	0.59237	0.49003
2	Phoenix	0.53378	0.53601	0.53156
3	BCAmirs	0.52613	0.55311	0.50166
4	CLaC	0.51401	0.49614	0.53322
5	27	0.49915	0.51408	0.48505
6	Baseline	0.48649	0.65000	0.38870
7	NLPNCHU	0.48323	0.59466	0.40698
8	BDA	0.41635	0.38248	0.45681
9	JM	0.37622	0.45519	0.32060
10	BERTastic	0.00000	0.00000	0.00000
10	HierarchyEverywhere	0.00000	0.00000	0.00000
10	SuteAlbastre	0.00000	0.00000	0.00000

Arabic - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	JM	0.62652	0.75000
2	BDA	0.62137	0.65000
3	BCAmirs	0.61487	0.63125
4	SheffieldVeraAI	0.61031	0.61250
5	BERTastic	0.60276	0.60625
6	NLPNCHU	0.58518	0.59375
7	HierarchyEverywhere	0.56196	0.66875
8	MemeSifters	0.55654	0.69375
9	Snarci	0.55540	0.55625
10	Hidetsune	0.52825	0.54375
11	SuteAlbastre	0.50066	0.54375
12	nowhash	0.49831	0.53125
13	DUTIR938	0.46894	0.51875
14	IITK	0.46706	0.46875
15	LMEME	0.40940	0.42500
16	Baseline	0.22705	0.29375
16	CVCoders	0.22705	0.29375

Note that the leaderboard is updated every 20 minutes, therefore your submission may not be listed right away.

Contact

twitter.com/_proppy_
Register to the google group of the task to ask questions and interact with other participants

SemEval 2024 shared task on "Multilingual Detection of Persuasion Techniques in Memes"

Leaderboard

Official SemEval 2024 task 4 Test Leaderboards

SemEval 2024 task 4 Dev Leaderboards (until Feb. 1 2024)

Post SemEval Leaderboards on the Test set (Work in Progress)

English - Subtask 1

English - Subtask 2a

English - Subtask 2b

Bulgarian - Subtask 1

Bulgarian - Subtask 2a

Bulgarian - Subtask 2b

North Macedonian - Subtask 1

North Macedonian - Subtask 2a

North Macedonian - Subtask 2b

Arabic - Subtask 1

Arabic - Subtask 2a

Arabic - Subtask 2b

Contact