1
$\begingroup$

Typically sequence taggers (specifically, linear-chain CRFs) in NLP use sentence level information, i.e., for each word, we define feature functions that only depend on the tokens in the sentence. I have a sequence tagging problem where I am trying to use document level information.

It is a BIO chunking problem, where each token has a tag {B, I, O}. My observation is that certain tokens that have a frequency > a threshold in a document are important, i.e., if in a sentence they are tagged as B or I, in another sentence they should have a higher probability of being tagged as I. How can I encode this information in a linear chain CRF?

One idea is to include the frequency for the token as a feature in the CRF itself. That did not work well, so I am trying to find other solutions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.