Problem with Bayes theorem and bigram probabilities I'm working with Bayes’ Theorem, but I can't fix the numbers, and I don't know why. I have a very simple set of sequential events, grouping them into bigrams (sequential groups of two events):
$$ABBAB \mapsto (AB)(BB)(BA)(AB)$$
$P(A) = 2/5; P(B)=3/5$
$P(A|B)$ (bigram $BA$) $= 1/2 = 0.5$ (there are just two bigrams that start with $B$, one followed by $A$)
$P(B|A)$ (bigram $AB$) $= 2/2 = 1$ (there are just two bigrams that start with $A$, both followed by $B$)
$P(B|B)$ (bigram $BB$) $= 1/2 = 1$ (there are just two bigrams that start with $B$, one followed by $B$)
So, I understand that with Bayes’ theorem $P(A|B) P(B) = P(B|A) P(A)$. In this case, $0.5 \times 0.6 \neq 1 \times 0.4$. Where does the mistake arise?
 A: I think your confusion comes from an abuse/simplification of notation that people use with n-gram models. There are two different events here, and they shouldn’t be mixed up.
To keep them separate, I’m going to reintroduce the random variables into the equation. For example, $P(X_i = \text{A} \mid X_{i + 1} = \text{B})$. $X_i$ is the $i$th symbol in the sequence, and $X_{i+1}$ is the one that follows it. We make the usual stationarity assumption.
You’re using Bayes’s rule to get $P(\text{A} \mid \text{B})$ and finding that it differs from what you get for $P(\text{A} \mid \text{B})$ by counting. But the Bayes’s rule version is $P(X_i = \text{A} \mid X_{i + 1} = \text{B})$, and the counting version is $P(X_{i + 1} = \text{A} \mid X_i=\text{B})$. Therefore, it’s completely natural for the two numbers to be different. One is reasoning about the predecessor; the other is reasoning about the successor.

For reassurance that the one from Bayes’s rule  is different:
$$
\begin{align}
P(X_{i}=\text{A} \mid X_{i+1}=\text{B}) &= \frac{P(X_{i+1}=\text{B} \mid X_i = \text{A}) P(X_i=\text{A})}{P(X_{i+1}=\text{B})}
\end{align}
$$
The RHS quantities are all things that you counted.

As an extra bit of fun: the probability of your sequence should be the same whether you construct your n-grams in the forward or reverse direction.
If we count in the reverse direction to get $P(X_{i}=\text{A} \mid X_{i+1}=\text{B})$, it’s $\dfrac{2}{3}$. (See my comment about counting n-grams.)
If instead we use Bayes’s rule and the forward-direction counts, it’s $\frac{\frac{2}{2} \times \frac{2}{6}}{\frac{3}{6}}=\frac{2}{3}$—the same quantity, evaluated in a different way.
