Overfitting happens for model that is already regularized

Question

I am now working on a binary text classification task. The model I am using is as follows

• Feature: universal sentence encoder (USE). USE is a feature extractor that is widely used in obtaining representation for longer text like sentence (rather than words).
• Model: Logistic regression (with regularization hyperparameter $C$ optimized on the validation set).

However, I observe the following performance metric on train/val/test set.

SPLIT=train
              precision    recall  f1-score   support

           0       0.79      0.83      0.81      5215
           1       0.73      0.67      0.70      3560

    accuracy                           0.77      8775
   macro avg       0.76      0.75      0.76      8775
weighted avg       0.77      0.77      0.77      8775

SPLIT=val
              precision    recall  f1-score   support

           0       0.75      0.77      0.76       573
           1       0.68      0.65      0.66       427

    accuracy                           0.72      1000
   macro avg       0.71      0.71      0.71      1000
weighted avg       0.72      0.72      0.72      1000

SPLIT=test
              precision    recall  f1-score   support

           0       0.79      0.50      0.61      1604
           1       0.53      0.81      0.64      1115

    accuracy                           0.63      2719
   macro avg       0.66      0.65      0.63      2719
weighted avg       0.68      0.63      0.62      2719

The machine learning 101 tells me the gap between train/test performance gap is probably due to overfitting. However, as this is a binary classification problem, the performance (even on training set) is far from satisfying. Therefore, my question is two-fold

• How do I alleviate overfitting for a model that is already regularized (like mine).
• What I could do to improve the performance, especially considering that the training set performance is not good. I know the go-to answer is to try more sophisticated model but what is the guideline for choosing the model?

Answer 1

1. The result on the test set is noisy due to the relatively small test set, so I'd not worry too much about small differences. However, you are right the drop is quite notable. It's worth double-checking how the training-validation-test split was done? E.g. are they truly from the same distribution, is there an information leak from training to validation set, but not to the test set?
2. You have a huge feature space and relative to that very little data. There's always the worry that your hyperparameter choice might in a sense overfit the validation set. The drop in performance as you go train -> val -> test is noticable, so it could be a case of the model being able to substantially overfit the training set, less so the validation set and not the test set. L2-regularized logistic regression is a problem with a convex loss function, so you'll get the true global loss minimum for your model (conceivably badly overfit). That may not truly be what one wants here. In constrast overparameterized neural networks trained with regularized SGD tend to not end up finding the sharpest global optima and finding wide and flat local optima seems to lead to better generalization.
3. I assume you are using scikit learn and your regularization is L2-regularization. You are picking the hyperparameter on a pretty small validation set, which might effectively be even smaller, if only a small amount of the data is anywhere near the decision boundary. You might be better off optimizing it with repeated-K-fold cross-validation instead of a single training-validation split. You also did not say how you pick the parameter, usually the + 1 SE rule helps. I don't think that's the main problem though.
4. You did not say what type of text you are working with, but a sentence encoder might not create a very good feature space here. The less good (and high-dimensional) the feature space, the more risk of overfitting to meaningless aspects.
5. A neural network (such as BERT or GPT-2 see e.g. here, but you could try with something simpler like ULMFiT) is usually a sensible thing to try for this problem. There's a bunch of ways of regularizing neural networks/improve their performance. Some typical ideas include: a) fine-tuning the language model to the training data before training the classifcation model, b) data augmentation (e.g. roundtrip translation like English->Spanish->English - huggingface has pre-trained translation models you could use for this, unless you have the budget to e.g. use the Google API), c) traditional regularization parameters (e.g. weight decay, drop-out), d) some sensible kind of learning rate schedule (e.g. the one-cycle policy or the flat-cosine schedule; either after picking a sensible learning rate using the learning rate finder) and/or early stopping. It's really worth looking at what highly placed solutions in Kaggle competitions used, people there have to worry a lot about their test set performance.

Answer 2

Your dataset is far too small for split sample validation to be reliable. Use a resampling method such as the bootstrap or 100 repeats of 10-fold cross-validation. You are using logistic regression, which is a probability model, but then you are discarding the probabilities and reverting to a classification system, and using discontinuous accuracy scores that will be fooled by a bogus model. For more see this. For better modeling and validation strategies related to probability models see RMS.

Answer 3

1. As @Dave suggests, train first a model that fits the training data. There is no point to talk about overfitting if the model does not fit the data at all. Weighted f1-score of 77% on binary classification is probably not what you would be happy with.

2. Such a model may be a random forest. Give a try to xgboost.

3. Think about regularisation when you are already fine on training data.

4. Poor test score may be caused by different distribution of the test data, not necessarily by overfitting. It seems that your validation set is similar to train set and there the gap is much smaller.