Larger ngrams vs nested ngrams? Whenever I see people using ngrams - I often see them looking for higher chains like 4-grams, 5-grams and so on. However, I'm wondering why I never see any mention of "nested ngrams" (I'm not sure what the real name is).
For example, suppose you tracked the bigram "to follow" followed by the bigram "after dark" instead of the 4-gram "to follow after dark".
Is there any reason to look into checking multiple nested levels of smaller ngrams instead of continuing to grow larger and larger ngram chains? It seems that one benefit might be storage costs.
 A: There is a somewhat similar idea, called a back-off language model. For these, you set a threshold $k$ and count up the number of n-grams in your training data. For n-grams that occur at least $k$ times, you estimate their probability in the normal way. Otherwise, you "back off" to the (n-1) gram's probability. I'm pretty sure there's a bit about this in Manning and Schutz's NLP book. People (or at least I) generally think about this as a way of avoiding over-fitting the training data, rather than representing any deep linguistic wisdom though.
There's also the "varigram" approach, where the context length (i.e., the $n$ in $n$-gram) isn't fixed. This might be a little closer to your/Henry's suggestion, since it would let you capture collocations/stock phrases/etc of different lengths. I think the idea is at least partly due to Kneser (1996). 
I'm not sure if your approach will save much storage space, either. For an $N$ word vocabulary, there are $N^2$ possible bigrams, $N^3$ possible trigrams, and $N^4$ quad-grams. Under your "paired bigram" model, there are still $(N^2) \cdot (N^2)=N^4$ possible pairs, though the sparseness might be a bit different. You may be better off storing the models as a (pruned) trie or something, if space is at a premium. 
