I have the text classification problem. Dataset is imbalanced in terms of classes. I'm using StratifiedKFold and balanced weights updating during training LogisticRegression. Let's say my score is:

  • 90% train
  • 60% validation

obviously, I need to address the overfitting, adding the regularization to the model. I would suspect that adding regularization will give something like:

  • 80% train
  • 65% validation

What I don't understand is that adding more regularization worsens validation score:

  • 80% train
  • 58% validation

On the other hand, reducing regularization gives:

  • 98% train
  • 62% validation

In theory, more regularization worsens train score but can help with the validation score. What may be the source of such behavior of the model?

  • $\begingroup$ 58 60 62 seems pretty close for me, especially when, as you say, class distribution is imbalanced. So my guess is that there is a much bigger problem hidden there than regularization: 1) how much testing example do you have ? 2) what is the proportion of the largest class ? May be in all your experiments, your model is overfitting, and there is no real difference in terms of generalization between your 3 models. $\endgroup$
    – xtof54
    Aug 9 '18 at 19:43
  • $\begingroup$ Thank you. I have 5k examples, I'm using Stratified cross-validation with 5 folds. There are 50 classes, some have ~25 examples, some ~200, one is much bigger - 600 $\endgroup$
    – DavidS1992
    Aug 10 '18 at 7:54
  • $\begingroup$ have you tried using class_wieghts or sample_weights? $\endgroup$ Aug 11 '18 at 6:24
  • $\begingroup$ Yes I'm using class_wieghts='balanced' in scikit $\endgroup$
    – DavidS1992
    Aug 12 '18 at 12:47

In your case, the Wald confidence interval gives about +/-1.36% on the evaluation set. So I'm not so sure whether you may really conclude that regularization negatively impacts your accuracy. I think all we can say is that it does not improve generalization. Why ? I'm sorry, but I don't know, given the information provided: many things may go wrong. For instance, I have the feeling that 50 classes is a lot for a total of 5k samples; may be try and analyze the accuracy per class, to get an idea about what's going on.

Good luck !

  • 1
    $\begingroup$ Rather than accuracy, precision-recall would be a better choice I think $\endgroup$ Aug 11 '18 at 6:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.