Skip to main content
2 of 2
grammar/readability
fnl
  • 1.6k
  • 12
  • 15

(1) That simply depends on the applied evaluation strategy, which should match your application requirements; Strict evaluations (as used, e.g., for the NER tasks in BioCreative) indeed require you find the exact cutoff, and would count your example as both a FP and FN.

(2) Because by only extracting a sub-string, it can become impossible to link the detected mention to the entity's identifier. Just imagine there might be several "Bank[s] of Chicago". (Which therefore explains why we [also] use the strict evaluation mode in BioCreative, in the first place.)

Therefore, depending on your use-case(s), you rather might want to evaluate the following:

  • your NER system's ability to correctly group the detected strings by entities (if you need to report all mentions found),
  • if each mention for a unique entity in the document has a non-strict labeling overlap with the relevant NER detections (if you need to detect/highlight all mentions for your use-case),
  • and/or that at least one instance in the string set extracted for a given entity can be properly linked to the entity's official identifier, i.e., your system at least once extracted "First Bank of Chicago" (if you need to link the mentions to some kind of identifier).

This is a far more use-case appropriate approach that we use today (in industry), but I am not sure you can find an academic paper doing NER evaluations that way. But it nicely explains why you might need both strict and non-strict label detection evaluations.

Also, note that most NER evaluation strategies/corpora do not take coreferences ("The bank said...") into account...

fnl
  • 1.6k
  • 12
  • 15