Question about the probability chain rule

Question

I've understood from this: Is this a correct statement of the probability chain rule? that in the chain rule for probability, conditioning can be done on different variables.

I was wondering what sort of implications this has for language modeling where we try and assign a probability to a sequence of m words P(w_1, w_2, ..., w_m) or any sequence for that matter.

For example, say we want to assign a probability to P(the, cat, ran).

As seen here: http://en.wikipedia.org/wiki/Language_model#N-gram_models one would compute P(the, cat, ran) by calculating P(the)*P(cat|the)*P(ran|the,cat).

But given that conditioning can be done on different variables, would it be correct to say that P(the, cat, ran) = P(ran)*P(the|ran)*P(cat|ran,the)?

If so, how is the sequence of the words actually taken into account?

The probability of "the cat ran" is obviously more likely than "ran the cat".

Answer 1

In standard notation, P(the, cat, ran) does not imply a particular ordering to the words. It simply indicates that three random variables have been observed taking those values.

Now, if you write the whole thing out in full, enforcing sentence ordering, you get

$$ P(w_t = \text{the}, w_{t+1} = \text{cat}, w_{t+2} = \text{ran}) $$

where $w_t$ indicates the word observed at time period $t$, $w_{t+1}$ the word observed at the next time period, and so on, then an ordering is enforced. Note that you can still condition on this too. So, $$ P(w_t = \text{the})P(w_{t+1} = \text{cat}|w_t = \text{the})P(w_{t+2} = \text{ran}|w_t = \text{the}, w_{t+1} = \text{cat}) $$ is the same as $$ P(w_{t+2} = \text{ran})P(w_t = \text{the}|w_{t+2} = \text{ran})P(w_{t+1} = \text{cat}|w_{t+2} = \text{ran}, w_t = \text{the}) $$ and there's no ambiguity in word order.

It's worth pointing out, though, that many models throw away the word ordering information, and treat a sentence as a "bag of words" in no particular sequence, for computational reasons. Under this model, yes, ran the cat is treated exactly the same as the cat ran - but that's a modeling choice made by the practitioner, not a fundamental (weird) property of the chain rule.