Typically sequence taggers (specifically, linear-chain CRFs) in NLP use sentence level information, i.e., for each word, we define feature functions that only depend on the tokens in the sentence. I have a sequence tagging problem where I am trying to use document level information.

It is a BIO chunking problem, where each token has a tag {B, I, O}. My observation is that certain tokens that have a frequency > a threshold in a document are important, i.e., if in a sentence they are tagged as B or I, in another sentence they should have a higher probability of being tagged as I. How can I encode this information in a linear chain CRF?

One idea is to include the frequency for the token as a feature in the CRF itself. That did not work well, so I am trying to find other solutions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.