I am looking at plots of learning curves (accuracy vs training examples) in order to compare different feature extraction methods that I am trying using a bag of words model (term presence vs. term frequency vs. tf-idf) for binary text classification. I am using ShuffleSplit with 10 splits for cross-validation using GridSearchCV.
I understand that generally speaking when cross-validation error decreases as the training set size increases, and there is a large gap between training error and cross-validation error, you have overfitting (high variance). A larger training set and reduced number of features could help in that case. Similarly when there is no gap between training and cross-validation error curves and still the training error is high, you have high bias, and trying to use a larger set of features may help.
This answer explains,
"Overfitting does not refer to the gap between training and test error being large or even increasing. It might be true that both training and testing error are decreasing, but training error is decreasing at a faster rate.... Overfitting specifically relates to the training error decreasing at the expense of model generalization (approximated through cross validation) as model hyperparameters are tuned..."
Given that explanation, it seems to me that this below plot shows overfitting. Other learning curves for additional models I am testing look more like the "traditional" overfitting that I describe in the previous paragraph.
With that in mind, I am wondering: At what point are your curves acceptable in a bag of words problem? Is it that they're acceptable once you achieve high precision / recall for your cross-validation and test sets?
My dataset consists of small, 1-6 word "documents" that are almost always noun phrases, and I have removed stop words, made all text lowercase, etc. in addition to trying feature selection with chi squared.
Overfitting can happen when you have features that cause learning models to memorize the training data, rather than generalize, which seems to me to be an inherent challenge that bag of words models will always face. In problems where we are using text data, to reference this question,
"Data sparsity is more of an issue in NLP than in other machine learning fields because we typically deal with large vocabularies where it is impossible to have enough data to actually observe examples of all the things that people can say. There will be many real phrases that we will just never see in the training data."
It makes sense then that more data would help, but when I am getting high accuracy/precision/recall on my test set, how do I know that my model is acceptable?
