SemEval 2025 Task 10
"Multilingual Characterization and Extraction of Narratives from Online News"

The internet has opened vast possibilities to easily create direct communication channels between information producers and consumers, potentially leaving the latter exposed to deceptive content and attempts at manipulation. Huge audiences can be affected online, and major crisis events are continuously subjected to the spread of harmful disinformation and propaganda.

In order to foster research and development of novel analytical functionalities to support end-users in analysing the news ecosystem and characterizing manipulation attempts, we launch in the frame of SemEval 2025 campaign a Task on Multilingual Characterization and Extraction of Narratives from Online News.

In particular, the task focuses on the automatic identification of narratives, their classification and identifying the roles of the relevant entities involved. These specific analytical dimensions are of paramount importance for facilitating the work of analysts studying target-specific disinformation phenomena.

We offer three subtasks on news articles: Entity Framing, Narrative Classification and Narrative Extraction, in five languages: Bulgarian, English, Hindi, Portuguese, and Russian. The participants may take part in any number of subtask-language pairs (even just one), and may train their systems using the data for all languages (in a multilingual setup).

The task builts on top of the prior tasks on media analysis that focused on persuasion techniques, framing dimensions and news genre organized as part of SemEval 2023, SemEval 2020, and CheckThat! 2024 Lab campaigns, and the task on detecting entity roles in memes organized as part of the CheckThat! 2024 Lab .

Technical Description

The task covers news articles from two domains, namely, Ukraine-Russia War and Climate Change, and is subdivided into three subtasks

Subtask 1: Entity Framing

Definition: given a news article and a list of mentions of named entities (NEs) in the article, assign for each such mention one or more roles using a predefined taxonomy of fine-grained roles covering three main type of roles: protagonists, antagonists, and innocent. This is a multi-label multi-class text-span classification task.

Subtask 2: Narrative Classification

Definition: given a news article and a two-level taxonomy of narrative labels (where each narrative is subdivided into subnarratives) from a particular domain, assign to the article all the appropriate subnarrative labels. This is a multi-label document classification task.

Subtask 3: Narrative Extraction

Definition: given a news article and a dominant narrative of the text of this article, generate a free-text explanation (up to max. of 80 words) supporting the choice of this dominant narrative. The to-be-generated explanation should be grounded in the text fragments that provide evidence of the claims of the dominant narrative. This is a text-to-text generation task.

Example

Below is an example of a news article focusing on climate change topics. The key entities and main claim-related text fragments are marked and underlined respectivelly.

Met Office Should Put 2.5°C ‘Uncertainties’ Warning on All Future Temperature Claims

It is “abundantly clear” that the Met Office cannot scientifically claim to know the cur- rent average temperature of the U.K. to a hundredth of a degree centigrade, given that it is using data that has a margin of error of up to 2.5°C, notes the climate journalist Paul Homewood. His comments follow recent disclosures in the Daily Sceptic that nearly eight out of ten of the Met’s 380 measuring stations come with official ‘uncertainties’ of between 2-5°C. In addition, given the poor siting of the stations now and possibly in the past, the Met Office has no means of knowing whether it is comparing like with like when it publishes temperature trends going back to 1884.

There are five classes of measuring stations identified by the World Meteorological Office (WMO). Classes 4 and 5 come with uncertainties of 2°C and 5°C respectively and account for an astonishing 77% of the Met Office station total. Class 3 has an uncertainty rating of 1°C and accounts for another 8.4% of the total. The Class ratings identify potential corruptions in recordings caused by both human and natural involvement. Homewood calculates that the average uncertainty across the entire database is 2.5°C. In the graph below, he then calculates the range of annual U.K. temperatures going back to 2010 incorporating the margins of error.

The blue blocks show the annual temperature announced by the Met Office, while the red bars take account of the WMO uncertainties. It is highly unlikely that the red bars show the more accurate temperature, and there is much evidence to suggest temperatures are nearer the blue trend. But the point of the exercise is to note that the Met Office, in the interests of scientific exactitude, should disclose what could be large measurement inac- curacies. This is particularly important when it is making highly politicised statements using rising temperatures to promote the Net Zero fantasy. As Homewood observes, the Met Office “cannot say with any degree of scientific certainty that the last two years were the warmest on record, nor quantify how much, if any, the climate has warmed since 1884”.

The U.K. figures are of course an important component of the Met Office’s global temper- ature dataset known as HadCRUT. As we noted recently, there is ongoing concern about the accuracy of HadCRUT with large retrospective adjustments of warming in recent times and cooling further back in the record. In fact, this concern has been ongoing for some time. The late Christopher Booker was a great champion of climate scepticism and in February 2015 he suggested that the “fiddling” with temperature data “is the biggest science scandal ever”. Writing in the Telegraph, he noted: “When future generations look back on the global warming scare of the past 30 years, nothing will shock them more than the extent to which official temperatures records – on which the entire panic rested – were systematically ‘adjusted’ to show the Earth as having warmed more than the actual data justified.”

The corresponding system response (in a simplified form here) for all three subtasks, i.e., entity roles, narrative classification and extraction, are provided below.

Entity Roles
Met Office: Antagonist-[Deceiver]
Paul Homewood: Protagonist-[Guardian]
Daily Sceptic: Protagonist-[Guardian]
Christopher Booker: Protagonist-[Guardian,Virtuous]

Narrative Classification: Questioning the measurements and science (narrative) - Methodologies/metrics used are unreliable/fault (subnarrative)

Narrative Extraction: Paul Homewood claims that the Met Office is misleading the public about current UK temperatures by not disclosing a margin of error of up to 2.5º C. The Daily Sceptic reports that most of the Met Office’s 380 stations provide inaccurate measurements. Additionally, Christopher Booker argues that official reports have been repeatedly falsified to indicate climate warming. Thus, the Met Office cannot conclude with scientific certainty that the climate is becoming warm

 article_id     entity_mention     start_offset     end_offset     main_role     fine-grained_roles
		

						
                               10001   Martin Luther King Jr.  10   32    Protagonist    Martyr
                               10002   Mahatma Gandhi          12   27    Protagonist    Martyr      Rebel
                               10003   ISIS                    4    8     Antagonist     Terrorist   Deceiver
						

 article_id     narrative_1;...;narrative_N     subnarrative_1;...;subnarrative_N     
	    

                              10001     Blaming Others                      Ukraine is the aggressor
                              10002     Blaming Others;Praise of Russia     Blaming Others-Other;Praising Russia’s military might
                              10003     Other                               Other
		

 article_id   dominant_narrative   dominant_subnarrative   explanation 
	    

 article_id   explanation 
	    

                                10001     Blaming Others     Ukraine is the aggressor            Ukraine war started...
                                10002     Praise of Russia   Praising Russia’s military might    Russia is...