3
$\begingroup$

I was watching Manning's lecture on evaluation of NER models, and I'm confused why between 4:07 and 5:13, he states that the error in not labeling one word in a sequence while correctly labeling the rest is considered as both a false positive (fp) and a false negative (fn). Based on my understanding, a fp is when you label a word as an entity when it is not actually that entity and a fn is when you don't label a word as an entity when it is truly is an entity. For those unable to watch the video, the example was:

Sentence: First Bank of Chicago
True label: ORG ORG ORG ORG
Model: O ORG ORG ORG

Questions:

  1. In the example, the model's prediction fails to label an entity that is truly an entity. Why is this not considered only a fp?
  2. Manning states that it is better not to have labeled any of the words as entities rather than naming part of the sequence correctly. Why is this true?
$\endgroup$
0
1
$\begingroup$

(1) That simply depends on the applied evaluation strategy, which should match your application requirements; Strict evaluations (as used, e.g., for the NER tasks in BioCreative) indeed require you find the exact cutoff, and would count your example as both a FP and FN.

(2) Because by only extracting a sub-string, it can become impossible to link the detected mention to the entity's identifier. Just imagine there might be several "Bank[s] of Chicago". (Which therefore explains why we [also] use the strict evaluation mode in BioCreative, in the first place.)

Therefore, depending on your use-case(s), you rather might want to evaluate the following:

  • your NER system's ability to correctly group the detected strings by entities (if you need to report all mentions found),
  • if each mention for a unique entity in the document has a non-strict labeling overlap with the relevant NER detections (if you need to detect/highlight all mentions for your use-case),
  • and/or that at least one instance in the string set extracted for a given entity can be properly linked to the entity's official identifier, i.e., your system at least once extracted "First Bank of Chicago" (if you need to link the mentions to some kind of identifier).

This is a far more use-case appropriate approach that we use today (in industry), but I am not sure you can find an academic paper doing NER evaluations that way. But it nicely explains why you might need both strict and non-strict label detection evaluations.

Also, note that most NER evaluation strategies/corpora do not take coreferences ("The bank said...") into account...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.