I am looking at plots of learning curves (accuracy vs training examples) in order to compare different feature extraction methods that I am trying using a bag of words model (term presence vs. term frequency vs. tf-idf) for binary text classification. I am using ShuffleSplit with 10 splits for cross-validation using GridSearchCV.

I understand that generally speaking when cross-validation error decreases as the training set size increases, and there is a large gap between training error and cross-validation error, you have overfitting (high variance). A larger training set and reduced number of features could help in that case. Similarly when there is no gap between training and cross-validation error curves and still the training error is high, you have high bias, and trying to use a larger set of features may help.

This answer explains,

"Overfitting does not refer to the gap between training and test error being large or even increasing. It might be true that both training and testing error are decreasing, but training error is decreasing at a faster rate.... Overfitting specifically relates to the training error decreasing at the expense of model generalization (approximated through cross validation) as model hyperparameters are tuned..."

Given that explanation, it seems to me that this below plot shows overfitting. Other learning curves for additional models I am testing look more like the "traditional" overfitting that I describe in the previous paragraph.

With that in mind, I am wondering: At what point are your curves acceptable in a bag of words problem? Is it that they're acceptable once you achieve high precision / recall for your cross-validation and test sets?

My dataset consists of small, 1-6 word "documents" that are almost always noun phrases, and I have removed stop words, made all text lowercase, etc. in addition to trying feature selection with chi squared.

Overfitting can happen when you have features that cause learning models to memorize the training data, rather than generalize, which seems to me to be an inherent challenge that bag of words models will always face. In problems where we are using text data, to reference this question,

"Data sparsity is more of an issue in NLP than in other machine learning fields because we typically deal with large vocabularies where it is impossible to have enough data to actually observe examples of all the things that people can say. There will be many real phrases that we will just never see in the training data."

It makes sense then that more data would help, but when I am getting high accuracy/precision/recall on my test set, how do I know that my model is acceptable?

Learning curves

  • $\begingroup$ Generally, accuracy plot is not the best metric to look at the performance of the model. This is because if the data is biased towards one class, the accuracy values will be high despite the model being only predicting one class at all times. I would suggest using the AUC (area under the curve) plot and the AUC value. $\endgroup$ – show_stopper Aug 23 '17 at 17:56
  • $\begingroup$ That's correct -- here is my test set classification report for reference: Negative class - Precision: 0.95, Recall: 1.00, F-score: 0.97, Support: 845. Positive class - Precision: 0.98, Recall: 0.85, F-score: 0.91, Support: 304. IIRC these numbers looked very similar when I set GridsearchCV to score based on roc_auc. I do have unbalanced classes though, and undersampled the negative class. $\endgroup$ – scmz Aug 23 '17 at 18:42
  • $\begingroup$ Given that you do use AUC curves and AUC value, if the training and cross validation curves are nearly the same (no matter how high or low the value is), it means that you have extracted the maximum capability out of the model. The only way to improve the performance is to either change the model or the feature set. Also there is no hard defined limit of how close the curves should be. It should be noted that the data in cross validation sets should be well shuffled while ensuring that the distribution of positive and negatives is same in all folds. $\endgroup$ – show_stopper Aug 24 '17 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.