Language model gives different results when Bayes' theorem is applied Say, the following example is our corpus:
a quick brown fox jumps over the lazy dog. 

Here, there are 9 tokens in the corpus. If we find the unigram probability for "a" it will be P("a") = 1 / 9 and also P("quick") = 1 / 9.
Similarly if we take the bigram probability of P("quick"|"a"):
P("quick"|"a") = CountOf(How many times "quick" followed by "a") / CountOf("a")
               = 1 / 1
               = 1

But for P("a"|"quick") it is 0.
Now we want to make an n-gram language model (n = 2) and want to find the joint distribution of the phrase "a quick", then according to the chain rule of probability:
P("a", "quick") = P("a")P("quick"|"a") = (1 / 9) * 1 = 1 / 9

Here, P("quick"|"a") means the probability of finding "quick" after "a".
But from Bayes' theorem we know:
P("a")P("quick"|"a") = P("quick")P("a"|"quick") = (1 / 9) * 0 = 0

But P("quick")P("a"|"quick") will give us 0 as P("a"|"quick") is 0. Whereas, P("a")P("quick"|"a") is not 0.
But mathematically they should be equal, but it is not happening. Moreover no matter how large corpus we take, the formulation found from Bayes' theorem will unlikely be exactly equal.
How we can describe this behavior? Is joint distribution order sensitive?
 A: I'm turning my comments into an answer. Because this question shows up in the introductory natural language processing class I TA, I'm answering in the style for a self-study question. I hope any other answerers will respect this and do the same.

The notation is tripping you up here.
You need to be extremely explicit about what $p(a \mid b)$ means. Specifically, you need to clearly state what the random variables involved are: $P(A = a \mid B = b)$; clearly identify $A$ and $B$. By abbreviating the notation, you've obfuscated a crucial distinction.
You're on the right track, and you need to go one step farther: What are the random variables? Take a look starting at slide 20 of these course notes on probability from the NLP class at Johns Hopkins.

A final hint: consider representing the sentence as a sequence of random variates. Now, the probability distribution over sentences is a joint distribution:
$$
P(X_1, X_2, X_3, X_4, \ldots)
\text{.}
$$
How can we tie this to the $P(\text{"a"})$ and $P(\text{"quick"} \mid \text{"a"})$ values you wrote above, using the hint I gave earlier in my answer?
