Evaluation metric for named entity recognition

Question

I was watching Manning's lecture on evaluation of NER models, and I'm confused why between 4:07 and 5:13, he states that the error in not labeling one word in a sequence while correctly labeling the rest is considered as both a false positive (fp) and a false negative (fn). Based on my understanding, a fp is when you label a word as an entity when it is not actually that entity and a fn is when you don't label a word as an entity when it is truly is an entity. For those unable to watch the video, the example was:

Sentence: First Bank of Chicago
True label: ORG ORG ORG ORG
Model: O ORG ORG ORG

Questions:

1. In the example, the model's prediction fails to label an entity that is truly an entity. Why is this not considered only a fp?
2. Manning states that it is better not to have labeled any of the words as entities rather than naming part of the sequence correctly. Why is this true?

Answer 1

(1) That simply depends on the applied evaluation strategy, which should match your application requirements; Strict evaluations (as used, e.g., for the NER tasks in BioCreative) indeed require you find the exact cutoff, and would count your example as both a FP and FN.

(2) Because by only extracting a sub-string, it can become impossible to link the detected mention to the entity's identifier. Just imagine there might be several "Bank[s] of Chicago". (Which therefore explains why we [also] use the strict evaluation mode in BioCreative, in the first place.)

Therefore, depending on your use-case(s), you rather might want to evaluate the following:

• your NER system's ability to correctly group the detected strings by entities (if you need to report all mentions found),
• if each mention for a unique entity in the document has a non-strict labeling overlap with the relevant NER detections (if you need to detect/highlight all mentions for your use-case),
• and/or that at least one instance in the string set extracted for a given entity can be properly linked to the entity's official identifier, i.e., your system at least once extracted "First Bank of Chicago" (if you need to link the mentions to some kind of identifier).

This is a far more use-case appropriate approach that we use today (in industry), but I am not sure you can find an academic paper doing NER evaluations that way. But it nicely explains why you might need both strict and non-strict label detection evaluations.

Also, note that most NER evaluation strategies/corpora do not take coreferences ("The bank said...") into account...

Answer 2

I'd like to quote from Speech and Language Processing: An introduction to natural language processing:

For named entities, the entity rather than the word is the unit of response.

In your case, the First Bank of Chicago should count as a single response, and it should be predicted as ORG ORG ORG ORG as a whole, otherwise the whole is wrong/false(either false positive or false negative).

If the predicted BIO tags are O B-ORG I-ORG I-ORG, it indicates a boundary error, and the whole is false and then O is false positive and B-ORG I-ORG I-ORG is false negative, two demerits.

However, if the guess tags are O O O O it is just a labeling error and there is only one demerit: one false positive.

In this article: Doing Named Entity Recognition? Don't optimize for F1, Christ Manning stated that the F1 encourages the model to guess all as O if it is not sure because boundary errors and label-boundary errors are more costly.

Side note:
An implementation of entity-level F1 score: https://github.com/jantrienes/nereval