News articles are inherently multimodal, combining text and images to convey information and shape readers' interpretations. While textual content provides explicit narratives, accompanying images play a crucial role in guiding attention and influencing how political figures and public events are perceived. Importantly, images can frame a news item in ways that differ from the framing expressed in the text. In this context, framing refers to selecting various salient aspects and perspectives of an item to convey a (possibly latent) meaning about an issue. In particular, images may reinforce, complement, or contradict textual framings. Together, these modalities give rise to a dimension that we refer to as multimodal framing, capturing how meaning emerges from the combination of text and images.
In order to foster research and development of novel analytical functionalities to support end-users in analyzing the news, in particular, enabling multimodal analysis of widely discussed events, cross-country comparisons of journalistic practices, media profiling and characterizing potential manipulation attempts, we launch in the frame of SemEval 2027 campaign a Task on Analysis of Multimodal Framing in the News.
In particular, the task focuses on the automatic identification and classification of framings in both text and images in news articles. These specific analytical dimensions are of paramount importance for facilitating the work of users in analyzing multimodal framing in the news ecosystem.
We offer two subtasks on news articles: Detection and Classification of Framings in Text, Detection and Classification of Framing in Images, in four languages: Bulgarian, English, (European) Portuguese, and Russian. The participants may take part in any number of subtask-language pairs (even just one), and may train their systems using the data for all languages (in a multilingual setup).
The task builts on top of the prior task on media analysis that focused on persuasion techniques, framing dimensions and news genre organized as part of SemEval 2023, and extends it by focusing on finer-grained, paragraph-level framing in text, which allows media analysts to more precisly identify and highlight evidence of specific framings within the text, and introducing a different modality, i.e., image framings.
Technical Description
The task covers news articles from various sources, inluding both mainstream media and unverified sources, and cover various topics.
Subtask 1: Framing in Text
Definition: given a text of a news article segmented into paragraphs, the goal is to identify zero or more framings conveyed by each paragraph in the text from a predefined taxonomy of 14 generic framings introduced in [Card et al. 2015] and listed HERE. This is a multi-label text classification task.
Subtask 2: Framing in Images
Definition: Given the text of a news article and an accompanying image, the goal is to identify zero or more framings conveyed by the image from a predefined taxonomy of framings (same as for Subtask 1). This is a multilabel image classification task. There are two variants of this task:
- Subtask 2a: no access to human-labeled (gold-standard) framings for the text is provided
- Subtask 2b: access to human-labeled (gold-standard) framings for the text is provided
Example
An input article, 2 images which could be associated with it, and corresponding annotations. Specific framing evidence in text is highlighted in blue.
Data Description
We will provide a training set to build your systems locally. We will further provide a development set (without annotations) and an online submission website to score your systems. A public leaderboard will show the progress on the task of the researchers involved in the task.
Input Articles
The input for all tasks will be news articles in plain text format in UTF-8. After registrations, participants will be able to download from their team page the corpus. Specifically, articles are provided in the folders train-articles-subtask-x. Further, we will provide a set of dev-articles-subtask-x for which annotations are not provided.
Each article appears in one .txt file. The title (if it exists) is on the first row, followed by an empty row. The content of the article starts from the third row
Articles in four languages (Bulgarian, English, (European) Portuguese, and Russian) were collected through exploiting various news aggregation engines, like for instance Europe Media Monitor (EMM), a large-scale multi-lingual near real-time news aggregation and analysis engine. The articles cover some widely-discussed topics, e.g., migration.
Gold Labels and Submission Format
Subtask 1 - Framing in Text
Ground truth
The format of the ground truth for each input document has the following format. Each line represents annotations for one paragraph, and contains the following elements separated by tabs:
start end framing-dimension
where start and end are the character offsets that serve as the boundaries of the paragraph, and framing-dimension is a list of zero or more framing dimensions assigned to the paragraph. The framing dimension labels are separated by tabs as well in case there is more than one. If there is no framing assigned to a given paragraph the field remains empty. This is an example of a section of the gold file for three paragraphs:
1307 2042 Policy_prescription_and_evaluation Economic Political
2046 2761 Policy_prescription_and_evaluation Economic Health_and_safety Quality_of_life
2765 3089 Economic Health_and_safety Quality_of_life
Partial view of a gold label file for Subtask 1
There is also file with an aggregation of all annotations, which has a similar format with an addition of the file names of the raw documents, i.e., using the following format:
file-name start end framing-dimension
System responses
For the testing phase the participants are provided with one single file with the following format. Each line corresponds to one paragraph in a given document, and contains the following elements separated by tabs:
file-name start end
The system response should constitute a single file with the same format as the ground truth (see above) using the following syntax.
file-name start end framing-dimension
In case no framing is detected in a given paragraph the framing-dimension field should remain empty.
Gold Labels and Submission Format
Subtask 2 - Framing in Images
Ground truth
The format of the file with the ground truth has the following format. Each line represents annotations for one image and document, and contains the following elements separated by tabs:
file-name framing-dimension
where file-name is the file with the image, and framing-dimension is a list of zero of more framing dimension labels, separated by tabs. framing-dimension remains empty if the image is not associated with any framing. This is an example of a section of the gold file for three images:
EN_TRIAL_5.png Public_opinion
EN_TRIAL_4.png Political
EN_TRAIL_2.png Legality_Constitutionality_and_jurisprudence Political
Partial view of a gold label file for Subtask 2
The image can be in any of the following formats: jpeg (jpg), png, webp, or gif.
System responses
For the testing phase the participants are provided with one single file which contains the file names (one by line) of the images for which framings should be detected. The system response should constitute a single file which has the following syntax, similar to the ground truth format (see above).
file-name framing-dimension
In case no framing is detected in a given image the framing-dimension field should remain empty.
Trial Data
Trial data consisting of several source articles in 4 languages and the corresponding ground truth annotation files are enclosed in this FILE
Evaluation
Upon registration, participants will have access to their team page, where they can also download scripts for scoring the different tasks. Here is a brief description of the evaluation measures the scorers compute.
Subtask 1
Subtask 1 is a multi-label multi-class classification problem. The official evaluation measure will be multi-label Micro F1. The corresponding Macro F1 will be computed as a complementary measure. Please note that we use the label None to represent no framing in text while computing the scores.
Subtask 2
Subtask 2 is a multi-label multi-class classification problem. The official evaluation measure will be multi-label Micro F1. The corresponding Macro F1 will be computed as a complementary measure. Please note that we use the label None to represent no framing in text while computing the scores.
For all subtasks, additional secondary evaluation scores (e.g., Macro F1) will be computed as well and displayed on the leaderboard.
Related datasets and material
Apart from the specific training datasets for our task, there are other related datasets that could be exploited in one way or another for assembling and elaborating models for both tasks:
- Media Framing Coprus, which covers multiple major policy issues across years of US news [Card et al., 2015]
- A corpus of 500K US news with texts and images automatically annotated with framings [Arora et al., 2025], which span various topics and political orientations
How to Participate
- Sign up for an account on Codabench HERE.
- Edit the profile of your account and add in the "Bio" field key information on all members of the team, including: first name, surname, affiliation and email. Provision of this information is required in order to participate in the task. Please also note that two teams can not share more than one member.
- Apply with the created account for one or more specific subtask-language combination you intend to participate HERE.
- Once the application is vetted (i.e., all participant information is provided) you can access the files tab to download the starter-kit and any training data available.
- Instructions on the submission of the system responses during the development and testing phase will follow shortly.
Dates
| 3 September 2027 | Task description available |
| 18 September 2027 | Registration opens |
| 10 January 2027 | Evaluation starts (to be detailed) |
| 31 January 2027 | Evaluation ends |
| February 2027 (tentative) | System paper submission deadline |
| March 2027 (tentative) | Notification to authors |
| April 2027 (tentative) | Camera ready papers due |
| Summer 2027 | SemEval workshop (co-located with a major NLP conference) |
Contact
To contact the organisers send a message to this email.
Credits
The following people are making this task possible (not exhaustive yet):
- Jakub Piskorski, Institute of Computer Science, Polish Academy of Sciences, Poland
- Dimitar Dimitrov, Sofia University, Bulgaria
- Giovanni Da San Martino, University of Padova, Italy
- Preslav Nakov, Mohamed bin Zayed University of Artificial Intelligence, UAE
- Yova Kementchedjhieva, Mohamed bin Zayed University of Artificial Intelligence, UAE
- Roman Yangarber, University of Helsinki, Finland
- Lidia Pivovarova, University of Helsinki, Finland
- Ricardo Campos, University of Beira Interior, Portugal
- Alípio Jorge, University of Porto and INESC TEC, Portugal
- Purificação Silvano, University of Porto, CLUP and INESC TEC, Portugal
- Nuno Ricardo Guimarães, University of Porto and INESC TEC, Portugal
- Ana Luísa Fernandes, University of Porto, INESC TEC and CLUP, Portugal
- Inês Cantante, University of Porto, CLUP, Portugal
- Rute Rebouças, University of Porto, CLUP and INESC TEC, Portugal
- Violeta Kastreva, ETH Zürich, Switzerland
- Elitsa Yotkova, Sofia University, Bulgaria
- Ivan Koychev, Sofia Univeristy, Bulgaria
Special Credits
We are greatly indebted to the authors of the work reported in [Arora et al., 2025] for sharing with us some human-labeled data that we exploit in the context of this shared task.