# Text classification - regularization worsens validation score

I have the text classification problem. Dataset is imbalanced in terms of classes. I'm using StratifiedKFold and balanced weights updating during training LogisticRegression. Let's say my score is:

• 90% train
• 60% validation

obviously, I need to address the overfitting, adding the regularization to the model. I would suspect that adding regularization will give something like:

• 80% train
• 65% validation

What I don't understand is that adding more regularization worsens validation score:

• 80% train
• 58% validation

On the other hand, reducing regularization gives:

• 98% train
• 62% validation

In theory, more regularization worsens train score but can help with the validation score. What may be the source of such behavior of the model?

• 58 60 62 seems pretty close for me, especially when, as you say, class distribution is imbalanced. So my guess is that there is a much bigger problem hidden there than regularization: 1) how much testing example do you have ? 2) what is the proportion of the largest class ? May be in all your experiments, your model is overfitting, and there is no real difference in terms of generalization between your 3 models. Aug 9 '18 at 19:43
• Thank you. I have 5k examples, I'm using Stratified cross-validation with 5 folds. There are 50 classes, some have ~25 examples, some ~200, one is much bigger - 600 Aug 10 '18 at 7:54
• have you tried using class_wieghts or sample_weights? Aug 11 '18 at 6:24
• Yes I'm using class_wieghts='balanced' in scikit Aug 12 '18 at 12:47