Overfitting in text classification task with word2vec

Question

I'm trying to complete a text classification task with word2vec, the steps I took are:

1. preprocess the text in my dataset;
2. split the dataset into training set(70%) and test set(30%);
3. train wrod2vec model with the text in the training set;
4. transfer all text(both in training and test set) into word2vec embeddings;
5. approach Naive Bayes, Logistic regression, and random forest to do the classification. RandomizedSearchCV was used to search for the optimal parameters.
6. use learning curves(use the data from the training set) to detect if the classifiers overfit or not.

The accuracy of all classifiers is similar, approx. 73%. However, all the learning curves of the classifiers showed that they were overfitted.

Below I give the learning curves:

Could anyone explain the situation? If there are some solutions for my problems?

Thanks in advance!

Answer 1

To add to gunes's answer, the score should improve as you get more and more samples if you have a proper model/estimate, one which increases its complexity as $n$, the sample size, grows. That is the whole point of statistics, to have consistent estimators as $n\to \infty$. In other words, the typical behavior should be that things improve when increasing $n$, if you have a good estimator.

For classification error, this roughly translates to this: For good estimators, your performance eventually approaches that of the optimal Bayes classifier (the best possible and the inherent information limit of the problem). In other words, the excess risk (your estimator's risk minus the Bayes risk) should go to zero in classification as $n \to \infty$ for reasonable estimators.

That the performance flattens out (say in the case of the random forest) is more a sign of underfitting. Your model's complexity reaches a limit and cannot incorporate more samples to improve, assuming that there is still room to improve (i.e., you are below that Bayes error rate). You can think about how in random forest one can increase the complexity (Hint: try keeping the number of samples you have at the leaves of the trees constant or not growing significantly with $n$ and see how it goes.)

Answer 2

The score may improve as you add more data, especially if it's not overfitting in the first place. In terms of overfitting, the final accuracies at the right end of the curves of the first two plots are quite close for test and training sets. I don't see a strong (or even weak) signal of overfitting in Logistic Regression and Naive Bayes models. The Random Forest needs some attention since the performances are not that close for the two sets.

Apart from everything, I'd also advise you to review this when considering performance metrics.