I'm working with Bayes’ Theorem, but I can't fix the numbers, and I don't know why. I have a very simple set of sequential events, grouping them into bigrams (sequential groups of two events):

$$ABBAB \mapsto (AB)(BB)(BA)(AB)$$

$P(A) = 2/5; P(B)=3/5$

$P(A|B)$ (bigram $BA$) $= 1/2 = 0.5$ (there are just two bigrams that start with $B$, one followed by $A$)

$P(B|A)$ (bigram $AB$) $= 2/2 = 1$ (there are just two bigrams that start with $A$, both followed by $B$)

$P(B|B)$ (bigram $BB$) $= 1/2 = 1$ (there are just two bigrams that start with $B$, one followed by $B$)

So, I understand that with Bayes’ theorem $P(A|B) P(B) = P(B|A) P(A)$. In this case, $0.5 \times 0.6 \neq 1 \times 0.4$. Where does the mistake arise?

  • 2
    $\begingroup$ If your values for P(A) and P(B) are to be consistent with your conditional probabilities, they should have denominator 4 rather than 5, since there are only 4 bigrams in your string. $\endgroup$
    – fblundun
    Commented Apr 5, 2021 at 18:25
  • $\begingroup$ I agree with fblundun that the counting is amiss here. Further, if this is a complete sequence (and not just a fragment), then there are really six bigrams, not four. These are ($\text{BOS}$, A), (A, B), (B, B), (B, A), (A, B), and (B, $\text{EOS}$) where $\text{BOS}$ and $\text{EOS}$ are special symbols to signal the beginning and end of the sequence. $\endgroup$ Commented Apr 5, 2021 at 18:29

1 Answer 1


I think your confusion comes from an abuse/simplification of notation that people use with n-gram models. There are two different events here, and they shouldn’t be mixed up.

To keep them separate, I’m going to reintroduce the random variables into the equation. For example, $P(X_i = \text{A} \mid X_{i + 1} = \text{B})$. $X_i$ is the $i$th symbol in the sequence, and $X_{i+1}$ is the one that follows it. We make the usual stationarity assumption.

You’re using Bayes’s rule to get $P(\text{A} \mid \text{B})$ and finding that it differs from what you get for $P(\text{A} \mid \text{B})$ by counting. But the Bayes’s rule version is $P(X_i = \text{A} \mid X_{i + 1} = \text{B})$, and the counting version is $P(X_{i + 1} = \text{A} \mid X_i=\text{B})$. Therefore, it’s completely natural for the two numbers to be different. One is reasoning about the predecessor; the other is reasoning about the successor.

For reassurance that the one from Bayes’s rule is different:

$$ \begin{align} P(X_{i}=\text{A} \mid X_{i+1}=\text{B}) &= \frac{P(X_{i+1}=\text{B} \mid X_i = \text{A}) P(X_i=\text{A})}{P(X_{i+1}=\text{B})} \end{align} $$

The RHS quantities are all things that you counted.

As an extra bit of fun: the probability of your sequence should be the same whether you construct your n-grams in the forward or reverse direction.

If we count in the reverse direction to get $P(X_{i}=\text{A} \mid X_{i+1}=\text{B})$, it’s $\dfrac{2}{3}$. (See my comment about counting n-grams.) If instead we use Bayes’s rule and the forward-direction counts, it’s $\frac{\frac{2}{2} \times \frac{2}{6}}{\frac{3}{6}}=\frac{2}{3}$—the same quantity, evaluated in a different way.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.