I am trying to understand if my results are overfitting or not. I have the following results, using different features for model building:

Model 1

Total classified: 4696
Score: 1.0 # from cross validation
Score length 3

Confusion matrix:
[[2348    0]
 [   0 2348]]

Logistic Regression 1
              precision    recall  f1-score   support

         0.0       0.96      0.97      0.97       585
         1.0       0.76      0.67      0.71        76

    accuracy                           0.94       661
   macro avg       0.86      0.82      0.84       661
weighted avg       0.94      0.94      0.94       661


Model 2

Total classified: 4696
Score: 0.65 # from cross validation
Score length 3

Confusion matrix:
[[2154  194]
 [  66 2282]]

    Logistic Regression 2
                  precision    recall  f1-score   support
             0.0       0.96      0.97      0.96       585
             1.0       0.73      0.68      0.71        76
        accuracy                           0.93       661
       macro avg       0.85      0.83      0.84       661
    weighted avg       0.93      0.93      0.93       661

It seems clear to me, looking at the model's 1 confusion matrix

   [[2348    0]
     [   0 2348]]

and at its score (1), that I am having a problem of overfitting. However, I would like to ask you the following questions, all related to this topic:

  • in the second model, I am getting a score of 65 and a confusion matrix not perfect. Would it be ok to say that it is not overfitting based on the other metrics in the contingency table (recall, f1 score,...) as they are not so far from that value? (the problem is a classification one, with imbalance data)
  • what about the accuracy in the contingency table?
  • is there anything else that I need to consider?

Thank you for all the answers and comments for clarifying this (challenging) concept.

  • $\begingroup$ Closely related stats.stackexchange.com/questions/312780/… $\endgroup$ – Sycorax Feb 23 at 3:49
  • $\begingroup$ Thank you Sycorax. I had not read that. However, it is not clear to me if the score I got in model 1 (or 2) is related to the accuracy in the classification report (or to another measure) for detecting overfitting. $\endgroup$ – Val Feb 23 at 19:35
  • $\begingroup$ If you divide a confusion matrix by the number of observations, the sum of the cells on the diagonal is exactly the accuracy of the model. A confusion matrix just displays the same information as accuracy, parceled out into different types of error. But it has the same weaknesses as accuracy, because it doesn’t tell you anything about the degree of error, so it has many of the same shortcomings as accuracy, too. $\endgroup$ – Sycorax Feb 23 at 19:38
  • $\begingroup$ thanks, Sycorax, it makes sense. I was wondering indeed how to spot overfitting in my case. In the first model, the perfect confusion matrix is giving me information on that. But in the second case, where I get a score of 0.65, I would say that it is not overfitting. Maybe I am wrong and I cannot say anything from the information that I have $\endgroup$ – Val Feb 23 at 19:47
  • $\begingroup$ What's the definition of overfitting? How would you use the information you have here to apply that definition? $\endgroup$ – Sycorax Feb 23 at 19:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.