0
$\begingroup$

I'm training a multi-class classifier on text documents using a very classic (and somewhat old-fashioned) method on a data set consisting in relatively long text documents (average of 3000 tokens). The 16 classes are somewhat unbalanced.

I do:

  1. stratified split into train/val/test sets: train and test set have respectively 120'000 and 30'000 examples.
  2. usual text preprocessing and cleaning;
  3. feature engineering: TF-IDF vectorizer on the 150'000 most frequent 1-2-3grams
  4. train a Linear SVM classifier
  5. evaluate on test set

I observe a significant overfitting: train accuracy = 93%, test accuracy = 73%.

In order to reduce the overfitting I add a feature selection step between steps 3 and 4 by selecting the 50'000 best features out of the 150'000 n-grams features using univariate feature selection (see https://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection). The performance is train accuracy = 80%, test accuracy = 73%, on the same train/test split than above.

We observe much less overfitting but the accuracy is the same although we should expect an increase of performance.
What does this mean? How do you interpret this behavior?

$\endgroup$
1
  • 1
    $\begingroup$ Just a comment: you have retained the accuracy of the model whilst reducing its complexity. IMO that is an improvement in the model. Additionally, it might have been that the features you discarded were irrelevant to prediction anyway $\endgroup$
    – jcken
    Commented Mar 15, 2021 at 14:37

1 Answer 1

0
$\begingroup$

Mmm

  • model 1: train accuracy = 93%, test accuracy = 73%.

  • model 2: train accuracy = 80%, test accuracy = 73%

Which model would you choose?

(Hard to tell without any more information on train/test sets, not even their size, but if the training set has a decent size and if we suspect that the test set may not cover all the domain, I would pick model 1).

Now, let's assume the Bayes optimal error is... 73% Whatever the model selection/regularization you choose, you will never get more than 73% on the testing set.

$\endgroup$
2
  • $\begingroup$ I edited the question to add a bit more information on the data set $\endgroup$
    – ivankeller
    Commented Mar 16, 2021 at 10:20
  • $\begingroup$ Regarding the relevant remark about the Bayes optimal error we know that we can do better than 73% because we tried with BERT-based model and got 78% accuracy. $\endgroup$
    – ivankeller
    Commented Mar 17, 2021 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.