Illustration of the Hierarchical Evaluation Function

The user makes two predictions: node 1.2 and node 2.1. The gold label is node 2.2

1. Given the prediction 1.2, the node and all of its ancestors are marked in yellow

2. Given the prediction 2.1, the node and all of its ancestors are marked in yellow together with the previous ones.

3. Given the gold label 2.2, the node and all of its ancestors are marked in green.

4. The intersection between the yellow and green nodes is computed (in orange).

The contribution of this example to the evaluation function is proportional to the number ratio of nodes in the intersection

w.r.t. the yellow nodes (precision) and w.r.t. the green nodes (recall).

Finally, the F1 between the two values is computed.

More details are available in section 6 of this paper.