5
$\begingroup$

How can I understand the label-bias problem in Hidden Markov Models? And why is CRF able to solve this problem?

Improve this question
$\endgroup$
1
  • $\begingroup$ This is an old question but an answer here might still be useful. Look at page 2 column 1 of the paper "Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data" by Lafferty et. al. It gives a single paragraph answer to this question. $\endgroup$
    – Pushpendre
    Commented May 26, 2015 at 15:28

4 Answers 4

Reset to default
2
$\begingroup$

Label-bias is not a problem for HMM,because input sequence is generated by the model. By global normalization, CRF model avoid this problem.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ This is too short for an answer, could you expand to make the answer useful? $\endgroup$
    – Xi'an
    Commented Nov 2, 2015 at 21:43
2
$\begingroup$

Based on Section 2 of "Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data" by Lafferty, J. et al,

I think it is "states with a single outgoing transition effectively ignores their observation. More generally, states with low-entropy next state distributions will take little notice of observations".

Honestly, I'm not quite sure if this is the Label Bias Problem. Because, I don't know why this is a problem. Aren't the next state distributions inferred from the training data? So low entropy next state distributions happen because that is what the data has..then the problem isn't the model.. it's the data's...

HTH.

Improve this answer
$\endgroup$
2
$\begingroup$

Suppose a simple finite state machine which was developed for named entity recognition.

In those kinds of machines, the states with a single outgoing transition effectively ignore their observation. In other words, the states with a single transition simply have to move to the next state without considering their current observation. More generally, states with low-entropy next state distributions will take little notice of observations.

Reference:

J. D. Lafferty, A. McCallum, and F. C. N. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data”, in Proceedings of the eighteenth international conference on machine learning, ser. ICML ’01, San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2001, pp. 282–289, ISBN: 1-55860-778-1. [Online]. Available: http://dl.acm.org/citation.cfm?id=645530.655813.

Improve this answer
$\endgroup$
1
$\begingroup$

CRF is a solution for MEMM and NOT for HMM. In markov model Label-bias is not a problem ,because input sequence is generated by the model (Farhana Liza). in MEMM, while calculating the transition probabilities, from every position (AKA state), the probabilities sums up to 1. So whats the problem? Lets say we have a state that is very UNlikely to happen, but when it does, with a very high probability (even 1) it would happen again. Now, if we have a long chain of states, there is a higher probability to stay in that position forever even though it is a state that is unlikely to happen!

In the CRF model we are using GLOBAL NORMALIZATION, which takes care of it and sums up all of the transition probabilities to 1.

Good luck!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.