I have an imbalanced binary classification problem. After all the preprocessing (scaling, feature selection), I am going through an hyperparameter optmimisation using GridSearchCV to find the best Logistic Classifier (I will try other models but for now I am focusing on LR). I use the recall score as a scoring because I need to reduce false negative. The result of this is the following best model:

Best parameters: 

 {'C': 0.02, 'penalty': 'l1', 'solver': 'liblinear'} 

Then I use this model to make the Precision and recall curve for in-sample and out-of-sample data, and the result is this:

In-Sample:: Logistic: f1=0.598 auc=0.752
Out-of-Sample:: Logistic: f1=0.781 auc=0.947

enter image description here

So my doubt is: the fact that the out of sample is quite better than in sample, is it only due to the fact that I am using regularization, which by definition introduces bias during the fit phase to better generalise (C=0.02, which means regularization parameter equal to 50)? or there is something strange here? I know, I put no code here (too long to post all), but I would just like to know if using regularization such result (out of sample better than in sample) is encountered sometimes..

  • $\begingroup$ What are the dataset sizes, did you do anything to adjust learning for imbalance, and how was the out-of-sample set chosen? $\endgroup$ – Ben Reiniger Jan 20 at 18:43
  • $\begingroup$ or the imbalanced, I used the option class_weight = 'balanced'. I also created a couple of dictionaries for weighing the class like {0:0.2, 1:1}..but I noticed that I was overfitting (recall=1 for train set and 0.70 test set), instead with the option 'balanced' the out-of-sample was better (recall=0.91 for train set and 0.96 test set). Train and test set have been splitted so that the ratio between the 2 classes is the same. Train set is 483 samples, test is 214 samples. $\endgroup$ – Luigi87 Jan 20 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.