I have a long sequence of words or letters {word1 word2 word3 word1 word1 word2 ..etc}. Lets say we extract all the ngrams (unigrams, bigrams, trigrams, 4-gram, 5-gram ....) along with their frequency from a text.

My question simply is how can we "statistically" find the best length of the n-gram given this text. is it unigrams, bigrams, trigrams, 4-gram, 5-gram or value-gram.
Note that I have no prior knowledge about the text.

  • 2
    $\begingroup$ What do you mean by "best n-gram"? Also, from a bag of words, how do you know a priori which words stand alone, which ones are paired, etc? $\endgroup$ – Antoine Jun 4 '15 at 14:51
  • 1
    $\begingroup$ It really depends on what you're trying to accomplish! If you're building a text classifier, you should perform cross-validation over various feature representations, and select your preferred approach considering your performance metric and domain knowledge. $\endgroup$ – Kyle. Jun 4 '15 at 15:14
  • $\begingroup$ What are you going to do with these n-grams? If you're just going to look at them, the best number is whichever looks prettiest =D. If you're building some kind of model the best number of n-grams is whatever gives your model the best performance on an out-of-sample dataset. $\endgroup$ – Zach Jun 4 '15 at 15:23
  • $\begingroup$ @ Zach: you mean that I should decide the best "n" after building my model by evaluating the performance of the whole model or system with different n; rather than estimating "n" before building the model. $\endgroup$ – Omar14 Jun 4 '15 at 15:29

There could be some statistical criteria for selecting the best $n$, but I believe the best way to select any parameter is to use cross-validation.

Suppose you're building a system for detecting the language of an article and you decided to use n-grams (n-shingles) to represent the documents. Then to select the best $n$, you split your dataset into training and testing subsets, and then run 10-fold cross-validation on the training set for each $n \in \{1, 2, 3, 4, \ ... \}$ and select such $n$ that minimizes the validation error.

CV can also be used in unsupervised learning, see e.g. (1)

[1] Patrick O. Perry, "Cross-Validation for Unsupervised Learning", http://arxiv.org/abs/0909.3052

  • $\begingroup$ Great! I'm trying to understand how cross validation works in the context of ngram models. Suppose we use k-cross validation. I understand that the model essentially lists the probability of each ngram from a text in training. However, how does cross validation work? What is the parameter that I should be estimate and compare? Any ideas? $\endgroup$ – Omar14 Jun 4 '15 at 15:13
  • $\begingroup$ I meant n-gram which is a contiguous sequence of n items from a given sequence of text or speech. e.g. "AABCV" ; suppose we have bigrams {AA,AB,BC,CV}. $\endgroup$ – Omar14 Jun 4 '15 at 16:07
  • $\begingroup$ Then the example should work for you $\endgroup$ – Alexey Grigorev Jun 4 '15 at 17:32
  • $\begingroup$ I was wondering if you could show a hands-on example of applying cross-validation in this situation $\endgroup$ – Andrew Bumetsov Sep 23 '20 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.