Sequence length when training a conditional random field (CRF) I am training a conditional random field (CRF) to perform named entity recognition. I have 1000 documents, each containing from 100 to 500 sentences.
During the training phase, is it better to train sentence per sentence, or document per document? I.e., as a sequence to the CRF, is it better to give a single sentence, or an entire document?
 A: You may want to use documents in two cases:


*

*you expect dependencies that cross sentence boundaries, e.g. if you think that last word in a previous sentence is helpful for tagging first word in next sentence.

*you want to avoid errors caused by imperfect sentence splitting - e.g. if a single named entity is incorrectly split into two parts by sentence splitter you've got an error which NER can't recover. Named entities are bad for sentence splitters because they often contain title-cased words and dots.


If your sentence splitter is perfect or near perfect than it makes sense to train on sentences - this will provide additional information for NER system in a form of 'hard boundaries'.
Another option is train on documents, but use sentence segmenter output as additional features for CRF.
There shouldn't be efficiency issues with passing documents to NER because training/prediction time is linear with respect to sequence length.
Documents are represented as larger matrices; it depends on CRF implementatin, but often it is more efficient to use a fewer number of larger matrices.
Using sentences/documents may affect optimization methods: if a form of SGD is used parameters will be updated faster with sentences because optimizer will be making more updates; also, by shuffling sentences you can get more randomized input than by shuffling documents; this can help SGD.
