SemEval 2024 shared task on "Multilingual Detection of Persuasion Techniques in Memes"

Official SemEval Results on the Test Set

English - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	914isthebest	0.75247	0.68419	0.83590
2	BCAmirs	0.69857	0.66786	0.73223
3	OtterlyObsessedWithSemantics	0.69738	0.64801	0.75490
4	TUMnlp	0.67384	0.63781	0.71419
5	GreyBox	0.66998	0.65248	0.68844
6	NLPNCHU	0.66271	0.60990	0.72552
7	Puer	0.66026	0.64833	0.67264
8	EURECOM	0.65529	0.62809	0.68495
9	SuteAlbastre	0.65234	0.63262	0.67334
10	UMUTeam	0.64774	0.70817	0.59681
11	RDproj	0.64288	0.57530	0.72846
12	HierarchyEverywhere	0.64252	0.63618	0.64899
13	nowhash	0.64096	0.61167	0.67320
14	ShefCDTeam	0.63951	0.66232	0.61821
15	Pauk	0.62660	0.71572	0.55722
16	IUSTNLPLAB	0.62479	0.63194	0.61780
17	whatdoyoumeme	0.61723	0.59769	0.63808
18	LomonosovMSU	0.61339	0.71154	0.53903
19	SoftMiner	0.60656	0.64947	0.56897
20	MagnumJUCSE	0.60341	0.54655	0.67348
21	IITK	0.59124	0.59624	0.58632
22	CLaC	0.57827	0.50056	0.68453
23	BAMBAS	0.57666	0.50124	0.67879
24	MemeSifters	0.57454	0.57619	0.57289
25	fralak	0.55701	0.47764	0.66802
26	IIITG	0.52569	0.61429	0.45943
27	Two	0.52197	0.52639	0.51763
28	Scalar	0.50481	0.43277	0.60562
29	SINAI	0.42519	0.31212	0.66676
30	McRock	0.42275	0.30141	0.70761
31	Baseline	0.36865	0.47711	0.30036
32	WhatsaMeme	0.34663	0.34728	0.34597
33	IIMAS1UTM1LaSalle	0.19917	0.75506	0.11472

English - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	HierarchyEverywhere	0.74592	0.86682	0.65461
2	NLPNCHU	0.70677	0.78164	0.64498
3	BCAmirs	0.70497	0.78374	0.64059
4	UMUTeam	0.69003	0.76763	0.62669
5	SuteAlbastre	0.68480	0.71777	0.65472
6	TUMnlp	0.67717	0.78066	0.59790
7	Pauk	0.67529	0.74503	0.61748
8	CodeMeme	0.66618	0.60655	0.73882
9	LomonosovMSU	0.65610	0.79153	0.56024
10	IITK	0.63605	0.76291	0.54537
11	BERTastic	0.61338	0.81581	0.49144
12	BDA	0.50388	0.51485	0.49337
13	Baseline	0.44706	0.68778	0.33116
14	WhatsaMeme	0.36592	0.31342	0.43955

English - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	LMEME	0.81030	0.82500
2	SuteAlbastre	0.80964	0.83500
3	DUTIR938	0.80910	0.83667
4	BCAmirs	0.80337	0.82500
5	Snarci	0.79860	0.82667
6	BDA	0.79290	0.82333
7	NLPNCHU	0.78803	0.82167
8	UMUTeam	0.78660	0.80667
9	TUMnlp	0.78413	0.80167
10	CodeMeme	0.78195	0.80667
11	LomonosovMSU	0.77226	0.79833
12	BERTastic	0.71582	0.76167
13	Hidetsune	0.71353	0.79000
14	Scalar	0.70151	0.75333
15	SheffieldVeraAI	0.64197	0.68667
16	HierarchyEverywhere	0.56309	0.66167
17	WhatsaMeme	0.51486	0.53000
18	nowhash	0.49845	0.51500
19	IITK	0.48336	0.49000
20	Baseline	0.25000	0.33333

Bulgarian - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	OtterlyObsessedWithSemantics	0.56833	0.51955	0.62722
2	RDproj	0.54089	0.43547	0.71367
3	NLPNCHU	0.51744	0.53578	0.50031
4	MagnumJUCSE	0.49986	0.47027	0.53342
5	nowhash	0.48627	0.46007	0.51563
6	MemeSifters	0.48125	0.49076	0.47210
7	GreyBox	0.47635	0.43821	0.52177
8	whatdoyoumeme	0.47257	0.50207	0.44635
9	HierarchyEverywhere	0.46757	0.48301	0.45310
10	fralak	0.46411	0.37360	0.61251
11	914isthebest	0.46309	0.47693	0.45003
12	CLaC	0.44917	0.40010	0.51196
13	BCAmirs	0.44834	0.38667	0.53342
14	IITK	0.43449	0.40422	0.46965
15	ShefCDTeam	0.36603	0.45413	0.30656
16	EURECOM	0.34483	0.36729	0.32495
17	SINAI	0.34118	0.21351	0.84856
18	Baseline	0.28377	0.31881	0.25567
19	SuteAlbastre	0.23573	0.13361	1.00000
20	IIMAS1UTM1LaSalle	0.18345	0.65414	0.10668

Bulgarian - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	BCAmirs	0.62693	0.70278	0.56586
2	SuteAlbastre	0.61074	0.65960	0.56862
3	NLPNCHU	0.54864	0.70691	0.44828
4	BERTastic	0.54358	0.81164	0.40862
5	Baseline	0.50000	0.80428	0.36276
6	BDA	0.48341	0.52265	0.44966
7	HierarchyEverywhere	0.46414	0.67080	0.35483
8	IITK	0.44589	0.54081	0.37931

Bulgarian - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	LMEME	0.67100	0.81000
2	Snarci	0.66777	0.84000
3	BERTastic	0.66212	0.75000
4	BCAmirs	0.64719	0.77000
5	NLPNCHU	0.64706	0.82000
6	MemeSifters	0.61143	0.83000
7	SuteAlbastre	0.59448	0.65000
8	SheffieldVeraAI	0.53619	0.57000
9	BDA	0.50624	0.62000
10	HierarchyEverywhere	0.48547	0.63000
11	IITK	0.47256	0.53000
12	DUTIR938	0.43414	0.57000
13	nowhash	0.43363	0.45000
14	Hidetsune	0.32670	0.33000
15	Baseline	0.16667	0.20000

North Macedonian - Subtask 1

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	OtterlyObsessedWithSemantics	0.51244	0.51824	0.50677
2	RDproj	0.49869	0.43364	0.58672
3	MagnumJUCSE	0.48267	0.48568	0.47970
4	fralak	0.46441	0.35882	0.65806
5	NLPNCHU	0.46165	0.54622	0.39975
6	EURECOM	0.44161	0.52000	0.38376
7	MemeSifters	0.44073	0.53915	0.37269
8	GreyBox	0.43435	0.43955	0.42927
9	nowhash	0.42558	0.41395	0.43788
10	HierarchyEverywhere	0.41713	0.48609	0.36531
11	CLaC	0.39471	0.37081	0.42189
12	BCAmirs	0.39298	0.33164	0.48216
13	IITK	0.38298	0.34412	0.43173
14	914isthebest	0.36860	0.40145	0.34071
15	whatdoyoumeme	0.36156	0.39852	0.33087
16	ShefCDTeam	0.31850	0.43590	0.25092
17	Baseline	0.30692	0.31403	0.30012
18	SINAI	0.30109	0.18312	0.84625
19	SuteAlbastre	0.20367	0.11343	0.99631
20	IIMAS1UTM1LaSalle	0.13704	0.52893	0.07872

North Macedonian - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	BCAmirs	0.63681	0.75019	0.55320
2	SuteAlbastre	0.57551	0.49247	0.69222
3	BERTastic	0.57329	0.86590	0.42849
4	Baseline	0.55525	0.90219	0.40103
5	BDA	0.50139	0.54619	0.46339
6	NLPNCHU	0.48707	0.70575	0.37185
7	IITK	0.43997	0.54476	0.36899
8	HierarchyEverywhere	0.35693	0.68903	0.24085

North Macedonian - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	BERTastic	0.68627	0.84000
2	MemeSifters	0.66033	0.90000
3	LMEME	0.59077	0.78000
4	BCAmirs	0.56098	0.77000
5	NLPNCHU	0.52000	0.79000
6	HierarchyEverywhere	0.50624	0.62000
7	IITK	0.48547	0.63000
8	Snarci	0.47917	0.72000
9	DUTIR938	0.46875	0.66000
10	SheffieldVeraAI	0.45790	0.51000
11	BDA	0.43535	0.60000
12	nowhash	0.42857	0.52000
13	Hidetsune	0.38942	0.46000
14	SuteAlbastre	0.17704	0.18000
15	Baseline	0.09091	0.10000

Arabic - Subtask 1

1	OtterlyObsessedWithSemantics	0.47593	0.39140	0.60702
2	NLPNCHU	0.47500	0.42817	0.53333
3	fralak	0.42796	0.30853	0.69825
4	whatdoyoumeme	0.42379	0.32759	0.60000
5	RDproj	0.41129	0.33333	0.53684
6	IITK	0.40782	0.33875	0.51228
7	HierarchyEverywhere	0.40545	0.35638	0.47018
8	nowhash	0.40370	0.35989	0.45965
9	BCAmirs	0.39625	0.32035	0.51930
10	MagnumJUCSE	0.39458	0.34565	0.45965
11	CLaC	0.38070	0.30803	0.49825
12	MemeSifters	0.35986	0.35495	0.36491
13	914isthebest	0.35982	0.31414	0.42105
14	Baseline	0.35897	0.35000	0.36842
15	SINAI	0.25770	0.15385	0.79298
16	SuteAlbastre	0.23429	0.19759	0.28772
17	EURECOM	0.17708	0.34343	0.11930

Arabic - Subtask 1 (Previous Version with Mistakes on Gold Labels)

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	NLPNCHU	0.47603	0.40563	0.57600
2	OtterlyObsessedWithSemantics	0.47399	0.37104	0.65600
3	RDproj	0.42031	0.32462	0.59600
4	IITK	0.41997	0.33179	0.57200
5	fralak	0.41341	0.28682	0.74000
6	whatdoyoumeme	0.40933	0.30268	0.63200
7	HierarchyEverywhere	0.40575	0.33777	0.50800
8	BCAmirs	0.40169	0.30952	0.57200
9	nowhash	0.39739	0.33516	0.48800
10	MagnumJUCSE	0.38156	0.31662	0.48000
11	CLaC	0.37693	0.29067	0.53600
12	Baseline	0.37091	0.34000	0.40800
13	914isthebest	0.34494	0.28534	0.43600
14	MemeSifters	0.32413	0.30034	0.35200
15	SINAI	0.23153	0.13547	0.79600
16	SuteAlbastre	0.22256	0.17831	0.29600
17	EURECOM	0.17765	0.31313	0.12400

Arabic - Subtask 2a

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	BCAmirs	0.52613	0.55311	0.50166
2	SuteAlbastre	0.51608	0.46939	0.57309
3	Baseline	0.48649	0.65000	0.38870
4	NLPNCHU	0.48323	0.59466	0.40698
5	IITK	0.45538	0.45729	0.45349
6	HierarchyEverywhere	0.43685	0.50998	0.38206
7	BDA	0.41635	0.38248	0.45681
8	BERTastic	0.38820	0.61290	0.28405

Arabic - Subtask 2a (Previous Version with Mistakes on Gold Labels)

Rank	Team	Hierarchical F1	Hierarchical Precision	Hierarchical Recall
1	Baseline	0.57002	0.64444	0.51101
2	BCAmirs	0.56200	0.51465	0.61894
3	SuteAlbastre	0.55341	0.44762	0.72467
4	NLPNCHU	0.51732	0.54369	0.49339
5	BERTastic	0.45293	0.59498	0.36564
6	IITK	0.43768	0.38526	0.50661
7	HierarchyEverywhere	0.43315	0.43459	0.43172
8	BDA	0.40921	0.33380	0.52863

Arabic - Subtask 2b

Rank	Team	F1 macro	F1 micro
1	BCAmirs	0.61487	0.63125
2	SheffieldVeraAI	0.61031	0.61250
3	BERTastic	0.60276	0.60625
4	NLPNCHU	0.58518	0.59375
5	HierarchyEverywhere	0.56196	0.66875
6	MemeSifters	0.55654	0.69375
7	Snarci	0.55540	0.55625
8	Hidetsune	0.52825	0.54375
9	BDA	0.50970	0.60625
10	SuteAlbastre	0.50066	0.54375
11	nowhash	0.49831	0.53125
12	DUTIR938	0.46894	0.51875
13	IITK	0.46706	0.46875
14	LMEME	0.36198	0.38750
15	Baseline	0.22705	0.29375

Note that the leaderboard is updated every 20 minutes, therefore your submission may not be listed right away.

Contact

twitter.com/_proppy_
Register to the google group of the task to ask questions and interact with other participants