We offer a shared task on the detection of Persuasion Techniques in multilingual online news. This task is part of the CheckThat! Lab 2024 edition and it is a follow-up of the SemEval 2023 Task 3, with respect to which it introduces some new elements.
The participants are provided with training data in various languages (see table below), i.e., news articles with Persuasion techniques. The task consists of building models capable of detecting 23 persuasion techniques at text-span level (see image) in news in English and four new languages: Portuguese, Slovenian, Bulgarian, Arabic.
Submissions may be made for any number of languages (even just one) and systems may be trained using any other annotated data available for the task. The data we provide is split into training, development and test sets, but the three sets will not be available for all languages (see table below). In the first phase we open a leaderboard for each language for which the development set is provided (the leaderboard will be updated in real-time). In the second phase we'll open a leaderboard for each language for which the test is provided (in this case the leaderboard will be visible only at the end of the test phase). Notice that the official ranking will be based only on the results on the test set, thus only on English, Arabic, Portuguese, Slovenian, Bulgarian.
Training set | Development set | Test set | |
English | X | X | X |
French | X | X | |
Italian | X | X | |
German | X | X | |
Russian | X | X | |
Polish | X | X | |
Spanish | X | ||
Greek | X | ||
Georgian | X | ||
Arabic | X | ||
Portuguese | X | ||
Slovenian | X | ||
Bulgarian | X |
All technical details about the task are given in the readme accessible from your team page after registering an account on this website. We share the annotation guidelines to give more detailed definitions, with examples, of the output classes for the task.
We provide a training set to build your systems locally. We further provide a development set (without annotations) and an online submission website to score your systems. A public leaderboard will show the progress on the task of the researchers involved in the task.
Upon registration, participants will have access to their team page, where they can also download scripts for scoring the task. Here is a brief description of the evaluation measures the scorers compute.
The task is a multi-label multi-class sequence tagging task. We modify the standard micro-averaged F1 to account for partial matching between the spans. In addition, an F1 value is computed for each persuasion technique. In a nutshell, strong partial overlap will be given a full credit (lean approach), while in cases in which the overlap is less strong the partial credit is proportional to the intersection of the two spans, and it is normalized by the length of the ground truth. The official score that will appear on Leaderboard will be computed using the 23 fine-grained persuasion technique labels. On top of this, an evaluation at the coarse-grained level will be computed too, i.e., mapping the labels to the 6 persuasion technique categories (see above) and this will be communicated to the participating teams.
We have created a google group for the task. Join it to ask any question and to interact with other participants.
Follow us on twitter to get the latest updates on the data and the competition!
If you need to contact the organisers only, send us an email.