6
$\begingroup$

I understand that discriminative models, such as CRF(Conditional Random Fields), model conditional probabilities $P(y|x)$, while generative models, such as HMM(Hidden Markov Model), model joint probabilities $P(y,x)$.

Take CRF and HMM for example. I know that CRF can have a larger range of possible features. Apart from that, what else makes CRF (discriminative models) preferable to HMM(generative models) in sequence labeling tasks such as Part-of-Speech tagging and NER(Name Entity Recognition)?

Edit:
I found out that HMMs will have to model $P(x)$, while CRFs don't. Why would it make a big difference in sequence labeling tasks?

$\endgroup$
  • 3
    $\begingroup$ It is a good idea to spell out acronyms and abbreviations. $\endgroup$ – Peter Flom Oct 16 '13 at 23:09
  • 1
    $\begingroup$ @PeterFlom Thanks for the suggestion, I've added the full spelling of some terms. $\endgroup$ – xiaoyao Oct 17 '13 at 0:06
  • 1
    $\begingroup$ This blog entry is one of the best and shortest intros to CRF I've ever read. After reading it, everything made a lot more sense to me. $\endgroup$ – jpmuc Nov 16 '13 at 16:29
  • 1
    $\begingroup$ ^That link is dead now. The blog entry: blog.echen.me/2012/01/03/… $\endgroup$ – abhshkdz Jan 29 '16 at 1:26
  • $\begingroup$ another mirror of the blog entry: web.archive.org/web/20120726150032/http://… $\endgroup$ – Franck Dernoncourt Jan 2 '17 at 18:05
2
$\begingroup$

I think you pretty much nailed it in your Edit. Generative model makes more restrictive assumption about the distribution of $x$.

From Minka

"Unlike traditional generative random fields, CRFs only model the conditional distribution $p(t|x)$ and do not explicitly model the marginal $p(x)$. Note that the labels ${ti }$ are globally conditioned on the whole observation $x$ in CRFs. Thus, we do not assume that the observed data $x$ are conditionally independent as in a generative random field."

$\endgroup$
2
$\begingroup$

CRFs and HMMs are not necessarily exclusive model formulations. In the formulation you have above, X in the HMM is usually a state variable that is unobserved, so a generative model is somewhat necessary. In the CRF though, X is some feature vector that is observed and affects Y in the traditional way. But you can have a combination of both: a sequence of states and outputs where the state is unobserved, and a set of observed features that affects the conditional probabilities of the outputs given the states (or transition probabilities between states).

I believe that ultimately the CRF admits some more flexible models where the conditional probabilities are more dynamic, and could be affected by, for example, the output from several observations ago, or something like that. They can get awfully large and difficult to train when they start including many more free parameters like that though.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.