3
$\begingroup$

why I am getting TP and FP in confusion matrix is 0, how to get it right? tp = 0,fp = 0,tn = 9847,fn = 18

$\endgroup$
3
  • $\begingroup$ Could you post your confusion Matrix and the prediction Error you are getting. A little bit more elaborated post should help. $\endgroup$ Mar 25, 2022 at 6:52
  • $\begingroup$ [9847, 0, 18, 0], $\endgroup$ Mar 25, 2022 at 7:12
  • $\begingroup$ You only have 18 true positive cases. How have you dealt with this imbalance? $\endgroup$
    – Firebug
    Mar 12, 2023 at 18:53

2 Answers 2

2
$\begingroup$

If you have no true positives or false positives, it means that your model only predicts negatives. Most machine learning models (such as logistic regressions and neural networks) work by making predictions about the probability of class membership.$^{\dagger}$ To get hard classifications, that predicted probability is compared to a threshold, typically $0.5$ as a software default, where cases are considered positive if their predictions are above the threshold and negative if their predictions are below the threshold.

Overall, it seems like your model is making a predictions of low probabilities of being a positive case. This need not be bad behavior by the model! It might be that positive cases are always unlikely. It might be that you want to use a different threshold, perhaps something closer to (or equal to) the prior probability of the minority class (that is, the proportion of minority class members). That way, you get alerted when there is a particularly high probability of the minority class occurring, even if that probability is fairly small. This is related to the idea that I discuss here. This is particularly common when the classes are imbalanced, such as in your case where you have almost $1000$ negative cases for every positive case, since the prior probability of minority class membership is so low.

However, it is not clear that you should be using a threshold of any kind, since the predicted probabilities are quite useful. Hopefully, some of the below links can convince you of this.

Cross Validated: Why is accuracy not the best measure for assessing classification models?

Cross Validated: Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

Cross Validated: Academic reference on the drawbacks of accuracy, F1 score, sensitivity and/or specificity

Cross Validated: Calculating the Brier or log score from the confusion matrix, or from accuracy, sensitivity, specificity, F1 score etc

Cross Validated: Upweight minority class vs. downsample+upweight majority class?

Cross Validated Meta: Profusion of threads on imbalanced data - can we merge/deem canonical any?

Frank Harrell's Blog: Classification vs. Prediction

Frank Harrell's Blog: Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

$^{\dagger}$ Neural networks tend to be overconfident in their predictions, so if a prediction of $p$ does not correspond to an event probability of $p$, then it is debatable if the predictions have a probabilistic interpretation. However, the predictions are on a continuum that can be thresholded to make hard classifications (if needed). Also, because the predicted probabilities are useful, the overconfident predictions by neural networks can be seen as problematic.

$\endgroup$
1
$\begingroup$

The confusion Matrix looks like this:

Confusion Matrix

Now, since you have only values popping for True Negative and False Negative, this would mean that in your TARGET feature, you mostly have One class values present in majority. For example:

Cancer/Non-Cancer detection

If in the above image you have only results popping for Non-Cancer, this could be because of the below reasons:

  1. Class - Imbalance : If so, try to even out the classes using sampling technique from sklearn.
  2. Overfitting : It is very much possible that due to imbalance, the model fails to learn pattern of one class and gets biased towards another class.

References: https://en.wikipedia.org/wiki/Confusion_matrix

Let me know if this helps.

$\endgroup$
3
  • $\begingroup$ thanks for your useful response, one more thing want to share, the above confusion matrix which i shared earlier had no tp and fp this was the logistic regression, but when i used random forest i got tp and fp , 6 and 31, it is ok or this model is not working. $\endgroup$ Mar 28, 2022 at 4:58
  • $\begingroup$ So, now you can see that as soon as you used a more extensive model, the RF was able to pick up some pattern, so yes definitely RF performed better than Logistic. I would recommend you to check for class imbalance and then try to approach the ML method. Upvote the answer if it helped.😄 $\endgroup$ Mar 28, 2022 at 5:06
  • 1
    $\begingroup$ It's not at all obvious that class imbalance is a problem that requires the modeler to mess with the data (such as with sklearn sampling techniques). Please refer to my answer and the links in it. $\endgroup$
    – Dave
    Mar 12, 2023 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.