I have a question about the label bias problem in HMM and CRF.

I understand that HMM and MEMM suffer from the label bias problem, which has preference over states with fewer transitions. The problem is due to the fact that in HMM and MEMM use local probabilities (each probability has to sum to 1), and thus, states with fewer transitions have a larger probability for its transitions. CRF alleviates this problem by global normalization, so that local potentials don't have to sum to 1.

What I am confused about is its relationship to I-equivalence between HMM/MEMM and CRF. Consider the following graphical models over variables $Y_{1:T}$ (where $X$ can be viewed as always given):

  • Directed graph $D$: $$Y_1\rightarrow Y_2\rightarrow\cdots\rightarrow Y_T $$
  • Undirected graph $G$: $$Y_1 - Y_2 - \cdots - Y_T $$

It's obvious that $D$ and $G$ have the same I-map, thus being I-equivalent.

In both cases, the inference objective is to find $$Y_{1:T}^*=\operatorname{argmax}_{Y_{1:T}} p(Y_1,\cdots, Y_T)$$ However, $D$ suffers from the lable bias problem, whereas $G$ doesn't.

I am wondering if $G$ has a stronger power of modeling joint probabilities than $D$ (i.e., $D$'s hypothesis class is a subset of $G$) despite their I-equivalence.

I would appreciate any insight. Thanks.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.