Leaderboard

Official SemEval 2024 task 4 Test Leaderboards

SemEval 2024 task 4 Dev Leaderboards (until Feb. 1 2024)

Post SemEval Leaderboards on the Test set (Work in Progress)

English - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	914isthebest	0.75247	0.68419	0.83590
2	BCAmirs	0.69857	0.66786	0.73223
3	OtterlyObsessedWithSemantics	0.69738	0.64801	0.75490
4	TUMnlp	0.67384	0.63781	0.71419
5	GreyBox	0.66998	0.65248	0.68844
6	Phoenix	0.66391	0.70398	0.62815
7	NLPNCHU	0.66271	0.60990	0.72552
8	Puer	0.66026	0.64833	0.67264
9	EURECOM	0.65702	0.65989	0.65417
10	HierarchyEverywhere	0.65286	0.63041	0.67697
11	SuteAlbastre	0.65234	0.63262	0.67334
12	ShefCDTeam	0.65088	0.70082	0.60758
13	UMUTeam	0.64774	0.70817	0.59681
14	RDproj	0.64288	0.57530	0.72846
15	nowhash	0.64096	0.61167	0.67320
16	whatdoyoumeme	0.63579	0.58301	0.69908
17	Pauk	0.62660	0.71572	0.55722
18	MAD2024	0.62541	0.58416	0.67292
19	IUSTNLPLAB	0.62479	0.63194	0.61780
20	CLaC	0.62161	0.57563	0.67557
21	IITK	0.61722	0.54040	0.71950
22	LomonosovMSU	0.61339	0.71154	0.53903
23	SoftMiner	0.60656	0.64947	0.56897
24	MagnumJUCSE	0.60341	0.54655	0.67348
25	CVCoders	0.59727	0.52563	0.69152
26	BAMBAS	0.58400	0.75946	0.47440
27	MemeSifters	0.57454	0.57619	0.57289
28	fralak	0.56521	0.48960	0.66844
29	IIITG	0.52569	0.61429	0.45943
30	Two	0.52197	0.52639	0.51763
31	Scalar	0.50481	0.43277	0.60562
32	SINAI	0.42519	0.31212	0.66676
33	McRock	0.42275	0.30141	0.70761
34	Baseline	0.36865	0.47711	0.30036
35	WhatsaMeme	0.34663	0.34728	0.34597
36	IIMAS1UTM1LaSalle	0.19917	0.75506	0.11472

English - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	HierarchyEverywhere	0.74592	0.86682	0.65461
2	NLPNCHU	0.70677	0.78164	0.64498
3	BCAmirs	0.70497	0.78374	0.64059
4	Phoenix	0.69666	0.73742	0.66018
5	SuteAlbastre	0.69508	0.67193	0.71988
6	UMUTeam	0.69003	0.76763	0.62669
7	TUMnlp	0.67717	0.78066	0.59790
8	Pauk	0.67529	0.74503	0.61748
9	IITK	0.67143	0.66954	0.67334
10	CodeMeme	0.66618	0.60655	0.73882
11	LomonosovMSU	0.65610	0.79153	0.56024
12	BERTastic	0.61338	0.81581	0.49144
13	BDA	0.50388	0.51485	0.49337
14	Baseline	0.44706	0.68778	0.33116
14	CVCoders	0.44706	0.68778	0.33116
15	WhatsaMeme	0.36592	0.31342	0.43955

English - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	LMEME	0.83637	0.86000
2	DUTIR938	0.81462	0.84000
3	SuteAlbastre	0.80964	0.83500
4	BCAmirs	0.80337	0.82500
5	Snarci	0.79860	0.82667
6	BDA	0.79290	0.82333
7	NLPNCHU	0.78803	0.82167
8	UMUTeam	0.78660	0.80667
9	TUMnlp	0.78413	0.80167
10	CodeMeme	0.78195	0.80667
11	LomonosovMSU	0.77226	0.79833
12	BERTastic	0.71582	0.76167
13	Hidetsune	0.71353	0.79000
14	Scalar	0.70151	0.75333
15	CVCoders	0.67544	0.70833
16	SheffieldVeraAI	0.64197	0.68667
17	HierarchyEverywhere	0.56309	0.66167
18	WhatsaMeme	0.51486	0.53000
19	nowhash	0.49845	0.51500
20	IITK	0.48336	0.49000
21	Baseline	0.25000	0.33333

Bulgarian - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	OtterlyObsessedWithSemantics	0.56833	0.51955	0.62722
2	RDproj	0.54133	0.43490	0.71674
3	NLPNCHU	0.51744	0.53578	0.50031
4	MagnumJUCSE	0.49986	0.47027	0.53342
5	nowhash	0.48627	0.46007	0.51563
6	IITK	0.48353	0.39586	0.62109
7	MemeSifters	0.48125	0.49076	0.47210
8	Phoenix	0.47695	0.49538	0.45984
9	GreyBox	0.47635	0.43821	0.52177
10	whatdoyoumeme	0.47257	0.50207	0.44635
11	HierarchyEverywhere	0.46757	0.48301	0.45310
12	fralak	0.46411	0.37360	0.61251
13	914isthebest	0.46309	0.47693	0.45003
14	SuteAlbastre	0.45318	0.43729	0.47026
15	CLaC	0.44917	0.40010	0.51196
16	BCAmirs	0.44834	0.38667	0.53342
17	ShefCDTeam	0.40283	0.44786	0.36603
18	EURECOM	0.34483	0.36729	0.32495
19	SINAI	0.34118	0.21351	0.84856
20	Baseline	0.28377	0.31881	0.25567
21	IIMAS1UTM1LaSalle	0.18345	0.65414	0.10668

Bulgarian - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	Phoenix	0.65638	0.67555	0.63828
2	IITK	0.64879	0.66414	0.63414
3	BCAmirs	0.62693	0.70278	0.56586
4	SuteAlbastre	0.61074	0.65960	0.56862
5	NLPNCHU	0.54864	0.70691	0.44828
6	BERTastic	0.54358	0.81164	0.40862
7	Baseline	0.50000	0.80428	0.36276
8	BDA	0.48341	0.52265	0.44966
9	HierarchyEverywhere	0.46414	0.67080	0.35483

Bulgarian - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	LMEME	0.67100	0.81000
2	Snarci	0.66777	0.84000
3	BERTastic	0.66212	0.75000
4	BCAmirs	0.64719	0.77000
5	NLPNCHU	0.64706	0.82000
6	MemeSifters	0.61143	0.83000
7	SuteAlbastre	0.59448	0.65000
8	BDA	0.57143	0.64000
9	SheffieldVeraAI	0.53619	0.57000
10	CVCoders	0.51637	0.74000
11	HierarchyEverywhere	0.48547	0.63000
12	IITK	0.47256	0.53000
13	DUTIR938	0.43414	0.57000
14	nowhash	0.43363	0.45000
15	Hidetsune	0.32670	0.33000
16	Baseline	0.16667	0.20000

North Macedonian - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	OtterlyObsessedWithSemantics	0.51244	0.51824	0.50677
2	RDproj	0.49894	0.43814	0.57934
3	MagnumJUCSE	0.48267	0.48568	0.47970
4	Phoenix	0.46615	0.62218	0.37269
5	fralak	0.46441	0.35882	0.65806
6	NLPNCHU	0.46165	0.54622	0.39975
7	EURECOM	0.44161	0.52000	0.38376
8	MemeSifters	0.44073	0.53915	0.37269
9	GreyBox	0.43435	0.43955	0.42927
10	nowhash	0.42558	0.41395	0.43788
11	HierarchyEverywhere	0.41713	0.48609	0.36531
12	IITK	0.41217	0.32662	0.55843
13	CLaC	0.39471	0.37081	0.42189
14	BCAmirs	0.39298	0.33164	0.48216
15	914isthebest	0.36860	0.40145	0.34071
16	whatdoyoumeme	0.36156	0.39852	0.33087
17	ShefCDTeam	0.35451	0.44362	0.29520
18	Baseline	0.30692	0.31403	0.30012
19	SINAI	0.30109	0.18312	0.84625
20	SuteAlbastre	0.27662	0.16785	0.78598
21	IIMAS1UTM1LaSalle	0.13704	0.52893	0.07872

North Macedonian - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	Phoenix	0.69844	0.76676	0.64130
2	IITK	0.66667	0.72465	0.61728
3	BCAmirs	0.63681	0.75019	0.55320
4	SuteAlbastre	0.62496	0.68130	0.57723
5	BERTastic	0.57329	0.86590	0.42849
6	Baseline	0.55525	0.90219	0.40103
7	BDA	0.50139	0.54619	0.46339
8	NLPNCHU	0.48707	0.70575	0.37185
9	HierarchyEverywhere	0.35693	0.68903	0.24085

North Macedonian - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	BERTastic	0.68627	0.84000
2	MemeSifters	0.66033	0.90000
3	LMEME	0.59077	0.78000
4	BDA	0.57765	0.74000
5	CVCoders	0.57653	0.79000
6	BCAmirs	0.56098	0.77000
7	NLPNCHU	0.52000	0.79000
8	HierarchyEverywhere	0.50624	0.62000
9	IITK	0.48547	0.63000
10	Snarci	0.47917	0.72000
11	DUTIR938	0.46875	0.66000
12	SheffieldVeraAI	0.45790	0.51000
13	nowhash	0.42857	0.52000
14	Hidetsune	0.38942	0.46000
15	SuteAlbastre	0.17704	0.18000
16	Baseline	0.09091	0.10000

Arabic - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	NLPNCHU	0.47500	0.42817	0.53333
2	Phoenix	0.44479	0.39510	0.50877
3	IITK	0.42637	0.35514	0.53333
4	whatdoyoumeme	0.42379	0.32759	0.60000
5	fralak	0.42117	0.30421	0.68421
6	RDproj	0.41454	0.33624	0.54035
7	BCAmirs	0.39625	0.32035	0.51930
8	CLaC	0.38070	0.30803	0.49825
9	Baseline	0.35897	0.35000	0.36842
10	914isthebest	0.00000	0.00000	0.00000
10	EURECOM	0.00000	0.00000	0.00000
10	HierarchyEverywhere	0.00000	0.00000	0.00000
10	MagnumJUCSE	0.00000	0.00000	0.00000
10	MemeSifters	0.00000	0.00000	0.00000
10	OtterlyObsessedWithSemantics	0.00000	0.00000	0.00000
10	SINAI	0.00000	0.00000	0.00000
10	SuteAlbastre	0.00000	0.00000	0.00000
10	nowhash	0.00000	0.00000	0.00000

Arabic - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	IITK	0.53636	0.59237	0.49003
2	Phoenix	0.53378	0.53601	0.53156
3	BCAmirs	0.52613	0.55311	0.50166
4	Baseline	0.48649	0.65000	0.38870
5	NLPNCHU	0.48323	0.59466	0.40698
6	BDA	0.41635	0.38248	0.45681
7	BERTastic	0.00000	0.00000	0.00000
7	HierarchyEverywhere	0.00000	0.00000	0.00000
7	SuteAlbastre	0.00000	0.00000	0.00000

Arabic - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	BDA	0.62137	0.65000
2	BCAmirs	0.61487	0.63125
3	SheffieldVeraAI	0.61031	0.61250
4	BERTastic	0.60276	0.60625
5	NLPNCHU	0.58518	0.59375
6	HierarchyEverywhere	0.56196	0.66875
7	MemeSifters	0.55654	0.69375
8	Snarci	0.55540	0.55625
9	Hidetsune	0.52825	0.54375
10	SuteAlbastre	0.50066	0.54375
11	nowhash	0.49831	0.53125
12	DUTIR938	0.46894	0.51875
13	IITK	0.46706	0.46875
14	LMEME	0.40940	0.42500
15	Baseline	0.22705	0.29375
15	CVCoders	0.22705	0.29375

Note that the leaderboard is updated every 20 minutes, therefore your submission may not be listed right away.

Contact

twitter.com/_proppy_
Register to the google group of the task to ask questions and interact with other participants

SemEval 2024 shared task on "Multilingual Detection of Persuasion Techniques in Memes"

Leaderboard

Official SemEval 2024 task 4 Test Leaderboards

SemEval 2024 task 4 Dev Leaderboards (until Feb. 1 2024)

Post SemEval Leaderboards on the Test set (Work in Progress)

English - Subtask 1

English - Subtask 2a

English - Subtask 2b

Bulgarian - Subtask 1

Bulgarian - Subtask 2a

Bulgarian - Subtask 2b

North Macedonian - Subtask 1

North Macedonian - Subtask 2a

North Macedonian - Subtask 2b

Arabic - Subtask 1

Arabic - Subtask 2a

Arabic - Subtask 2b

Contact