# Software packages that can construct feature representation for a given text file using N-gram model

Are there any open source software packages, (Java, Matlab, R) that can generate a feature representation for a text document using N-gram model?

-

vowpal wabbit can do that automatically (via --ngram flag) but it uses a highly convoluted way to store the resulting representation (a binary hash map) so it is not trivial to get a human-readable version of that.
If the data set in question is not enormous then one can construct ngram features with a bit of python scripting by relying on nltk package:
>>> help(nltk.ngrams) Help on function ngrams in module nltk.util: ngrams(sequence, n, pad_left=False, pad_right=False, pad_symbol=None) A utility that produces a sequence of ngrams from a sequence of items. For example:
>>> ngrams([1,2,3,4,5], 3) [(1, 2, 3), (2, 3, 4), (3, 4, 5)]
Use ingram for an iterator version of this function. Set pad_left or pad_right to true in order to get additional ngrams:
>>> ngrams([1,2,3,4,5], 2, pad_right=True) [(1, 2), (2, 3), (3, 4), (4, 5), (5, None)]