Classifier accuracy decreases as n of n-gram models increases. Is this expected? I am trying to tackle a multi-classification problem that requires text processing. The data contains a lot of samples (approximately 100.000 samples) and one of the features I need to work with is a short (sometimes longer) written message. Dividing the dataset into a 70/30 scheme and cross-validating on the 70% for tuning, the model reaches currently 90% accuracy. I am using scikit-learn's TfidfVectorizer to extract features from the messages and RandomForestClassifier for the machine learning per se. 
What I find peculiar is that as I increase the maximum n for the n-gram models considered, its accuracy decreases slightly :
n in [1,2[  ->    Accuracy: 91.0%   Dimensionality:   126,756
n in [1,2]  ->    Accuracy: 90.8%   Dimensionality:   605,010
n in [1,3]  ->    Accuracy: 90.5%   Dimensionality: 1,408,346
...         ->    ...

I initially thought that increasing the maximum n could only increase the model's accuracy but that is not reflected in my results.
Am I doing something wrong in my approach? Or is this decrease in accuracy something that could potentially happen?
 A: The problem with spurious variables in Random Forest is that each tree selects only a random subset of features, and if their number gets high, so if most of them are bogus (which is most likely the case here, since most of higher n-grams will occur only a bunch of times), some trees won't learn anything useful.
Random Forests are used because single decision tree is highly likely to overfit, and averaging predictions in decreases variance compared to using a single estimator. It doesn't seem likely that your single decision trees are overfitting, since they are using only a tiny random subset of huge, mostly irrelevant feature space.
You didn't say anything about the trees - did you try using more of them and simultaneously limiting their depth? Scikit-learn's Random Forests have a couple of parameters that you can tune.
Another question would be if you really actually need decision trees - I for example would at least try using logistic regression (with Lasso/ElasticNet) for such problem, as these methods naturally fit such sparse problems - they consider all the features, and do feature selection themselves.
A: These results are the same for character n-grams as well. until a certain n value, accuracy will increase and then it starts to drop for both word n-grams and character n-grams. Reason for this is if the individual accuracy of a given n value is lower for a particular selected corpus size it has a slightly negative effect on combined feature selection accuracy. (larger the n larger can be the corpus size needed for improving a unit of accuracy)
