3
$\begingroup$

I'm trying to complete a text classification task with word2vec, the steps I took are:

  1. preprocess the text in my dataset;
  2. split the dataset into training set(70%) and test set(30%);
  3. train wrod2vec model with the text in the training set;
  4. transfer all text(both in training and test set) into word2vec embeddings;
  5. approach Naive Bayes, Logistic regression, and random forest to do the classification. RandomizedSearchCV was used to search for the optimal parameters.
  6. use learning curves(use the data from the training set) to detect if the classifiers overfit or not.

The accuracy of all classifiers is similar, approx. 73%. However, all the learning curves of the classifiers showed that they were overfitted.

Below I give the learning curves:

enter image description here enter image description here enter image description here

Could anyone explain the situation? If there are some solutions for my problems?

Thanks in advance!

$\endgroup$
2
  • $\begingroup$ I don't think there is serious overfitting except the third one. $\endgroup$
    – gunes
    Jul 9 at 10:23
  • 1
    $\begingroup$ Thanks for your prompt reply! @gunes. For me, it is the first time to see the learning curves like these. I read the articles about diagnosing over/underfitting via learning curves, which said that low bias indicates that a classifier doesn't overfit. It seems my first two curves show that, but I have never seen an increasing training score before. And the training score and test score won't converge even I added more samples. May I still choose Naive Bayes and Logistic Regression based on their learning curves? $\endgroup$
    – BigTeeth
    Jul 9 at 10:43
4
$\begingroup$

To add to gunes's answer, the score should improve as you get more and more samples if you have a proper model/estimate, one which increases its complexity as $n$, the sample size, grows. That is the whole point of statistics, to have consistent estimators as $n\to \infty$. In other words, the typical behavior should be that things improve when increasing $n$, if you have a good estimator.

For classification error, this roughly translates to this: For good estimators, your performance eventually approaches that of the optimal Bayes classifier (the best possible and the inherent information limit of the problem). In other words, the excess risk (your estimator's risk minus the Bayes risk) should go to zero in classification as $n \to \infty$ for reasonable estimators.

That the performance flattens out (say in the case of the random forest) is more a sign of underfitting. Your model's complexity reaches a limit and cannot incorporate more samples to improve, assuming that there is still room to improve (i.e., you are below that Bayes error rate). You can think about how in random forest one can increase the complexity (Hint: try keeping the number of samples you have at the leaves of the trees constant or not growing significantly with $n$ and see how it goes.)

$\endgroup$
1
  • 1
    $\begingroup$ Here is some useful background on this (as I consider, correct) answer: machinelearningmastery.com/… - which implies the first two plots actually look good, while the last one is a sign of underfitting, not overfitting. (Note that the linked post uses loss, not accuracy, so you have to kind of mirror them to see how the LCs match to your case.) $\endgroup$
    – fnl
    Jul 15 at 1:26
2
$\begingroup$

The score may improve as you add more data, especially if it's not overfitting in the first place. In terms of overfitting, the final accuracies at the right end of the curves of the first two plots are quite close for test and training sets. I don't see a strong (or even weak) signal of overfitting in Logistic Regression and Naive Bayes models. The Random Forest needs some attention since the performances are not that close for the two sets.

Apart from everything, I'd also advise you to review this when considering performance metrics.

$\endgroup$
1
  • $\begingroup$ Thanks a lot! @gunes. Both your answer and your shared content are helpful! $\endgroup$
    – BigTeeth
    Jul 9 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.