# Estimating the best length of n-gram

I have a long sequence of words or letters {word1 word2 word3 word1 word1 word2 ..etc}. Lets say we extract all the ngrams (unigrams, bigrams, trigrams, 4-gram, 5-gram ....) along with their frequency from a text.

My question simply is how can we "statistically" find the best length of the n-gram given this text. is it unigrams, bigrams, trigrams, 4-gram, 5-gram or value-gram.
Note that I have no prior knowledge about the text.

• What do you mean by "best n-gram"? Also, from a bag of words, how do you know a priori which words stand alone, which ones are paired, etc? – Antoine Jun 4 '15 at 14:51
• It really depends on what you're trying to accomplish! If you're building a text classifier, you should perform cross-validation over various feature representations, and select your preferred approach considering your performance metric and domain knowledge. – Kyle. Jun 4 '15 at 15:14
• What are you going to do with these n-grams? If you're just going to look at them, the best number is whichever looks prettiest =D. If you're building some kind of model the best number of n-grams is whatever gives your model the best performance on an out-of-sample dataset. – Zach Jun 4 '15 at 15:23
• @ Zach: you mean that I should decide the best "n" after building my model by evaluating the performance of the whole model or system with different n; rather than estimating "n" before building the model. – Omar14 Jun 4 '15 at 15:29

There could be some statistical criteria for selecting the best $n$, but I believe the best way to select any parameter is to use cross-validation.
Suppose you're building a system for detecting the language of an article and you decided to use n-grams (n-shingles) to represent the documents. Then to select the best $n$, you split your dataset into training and testing subsets, and then run 10-fold cross-validation on the training set for each $n \in \{1, 2, 3, 4, \ ... \}$ and select such $n$ that minimizes the validation error.