# Problem with Bayes theorem and bigram probabilities

I'm working with Bayes’ Theorem, but I can't fix the numbers, and I don't know why. I have a very simple set of sequential events, grouping them into bigrams (sequential groups of two events):

$$ABBAB \mapsto (AB)(BB)(BA)(AB)$$

$$P(A) = 2/5; P(B)=3/5$$

$$P(A|B)$$ (bigram $$BA$$) $$= 1/2 = 0.5$$ (there are just two bigrams that start with $$B$$, one followed by $$A$$)

$$P(B|A)$$ (bigram $$AB$$) $$= 2/2 = 1$$ (there are just two bigrams that start with $$A$$, both followed by $$B$$)

$$P(B|B)$$ (bigram $$BB$$) $$= 1/2 = 1$$ (there are just two bigrams that start with $$B$$, one followed by $$B$$)

So, I understand that with Bayes’ theorem $$P(A|B) P(B) = P(B|A) P(A)$$. In this case, $$0.5 \times 0.6 \neq 1 \times 0.4$$. Where does the mistake arise?

• If your values for P(A) and P(B) are to be consistent with your conditional probabilities, they should have denominator 4 rather than 5, since there are only 4 bigrams in your string. Commented Apr 5, 2021 at 18:25
• I agree with fblundun that the counting is amiss here. Further, if this is a complete sequence (and not just a fragment), then there are really six bigrams, not four. These are ($\text{BOS}$, A), (A, B), (B, B), (B, A), (A, B), and (B, $\text{EOS}$) where $\text{BOS}$ and $\text{EOS}$ are special symbols to signal the beginning and end of the sequence. Commented Apr 5, 2021 at 18:29

I think your confusion comes from an abuse/simplification of notation that people use with n-gram models. There are two different events here, and they shouldn’t be mixed up.

To keep them separate, I’m going to reintroduce the random variables into the equation. For example, $$P(X_i = \text{A} \mid X_{i + 1} = \text{B})$$. $$X_i$$ is the $$i$$th symbol in the sequence, and $$X_{i+1}$$ is the one that follows it. We make the usual stationarity assumption.

You’re using Bayes’s rule to get $$P(\text{A} \mid \text{B})$$ and finding that it differs from what you get for $$P(\text{A} \mid \text{B})$$ by counting. But the Bayes’s rule version is $$P(X_i = \text{A} \mid X_{i + 1} = \text{B})$$, and the counting version is $$P(X_{i + 1} = \text{A} \mid X_i=\text{B})$$. Therefore, it’s completely natural for the two numbers to be different. One is reasoning about the predecessor; the other is reasoning about the successor.

For reassurance that the one from Bayes’s rule is different:

\begin{align} P(X_{i}=\text{A} \mid X_{i+1}=\text{B}) &= \frac{P(X_{i+1}=\text{B} \mid X_i = \text{A}) P(X_i=\text{A})}{P(X_{i+1}=\text{B})} \end{align}

The RHS quantities are all things that you counted.

As an extra bit of fun: the probability of your sequence should be the same whether you construct your n-grams in the forward or reverse direction.

If we count in the reverse direction to get $$P(X_{i}=\text{A} \mid X_{i+1}=\text{B})$$, it’s $$\dfrac{2}{3}$$. (See my comment about counting n-grams.) If instead we use Bayes’s rule and the forward-direction counts, it’s $$\frac{\frac{2}{2} \times \frac{2}{6}}{\frac{3}{6}}=\frac{2}{3}$$—the same quantity, evaluated in a different way.