I have no trouble understanding the definition of the TP, FP, TN, and FN, along with precision, recall, sensitivity, and the likes. I also understand that ROC-AUC is a graph that plots True Positive Rate against False Positive Rate, parametrized by a threshold of $T$. What I am confused about is depicted in the image below, we can clearly define the red line to be the positive class 1, and the green line to be the negative class 0, subsequently, the author mentioned (rephrased by me) that the green curve shows the distribution of the positive class and the red curve shows the distribution of negative class over the probability assigned to each obs by the classifier under a threshold of $T=0.5$.

Now I am not good in statistics, and by distribution, does the author mean the original distribution of the sample, or the distribution after the classifier predicts? In other words, assuming our ground truth data has 5 patients, and 2 is positive class, 3 is negative class; and our classifier outputs a softmax (preference) of probabilities like

y_preds = [[0.2, 0.8], [0.3, 0.7], [0.6, 0.4], [0.7, 0.3], [0.9, 0.1]]

Now since it's a binary problem, and by definition we should only look at the positive class's probabilities as the negative class's probabilities are derived automatically, the y_preds should be flattened to be:

y_preds = [0.8, 0.7, 0.4, 0.3, 0.1]

Back to out question, so is this graph depicting the distribution AFTER the classifier predicts, or before that? Furthermore, how is this graph plotted? Because to even be able to tell the FP and FN areas (the intersection), we definitely need to compare the predictions to the ground truth. Sorry if I did not make myself clear, but the graph is really throwing me off and would appreciate a more detailed explanation.

enter image description here

Edit 1: Ok I got it. So there are 100 patients in total and assume 70 benign and 30 malignant as the ground truth labels. Are we trying to say that we individually look at the predictions made for each group, say plot all 70 benign patients predictions on the graph and 30 malignant predictions on the graph with a threshold of T=0.5. A perfect classifier will then have all 70 benign predictions having a probability of less than 0.5 while 30 malignant predictions having a probability of more than 0.5 - the graphs will not intersect.

But if just one patient in the benign group has a prediction probability of 0.8 - that will classify it as malignant in our model - hence there will be an intersection, am I right with the analogy.

To confirm, the author said that the x axis is actually P(X=“positive class”) but that’s not the case here since it’s both?

  • $\begingroup$ The first paragraph of your edit is correct for now. You’ll go on to learn why accuracy is a surprisingly poor performance metric, but it’s fine to discuss now. // The second paragraph contains a mistake. If the threshold is set at 0.5, then that 0.8 will misclassified, but there will be no overlap of the other group has all predicted probabilities $>0.8$. // That author is correct. This is a histogram of predicted probabilities of being in the positive class. You happen to know which samples truly belonged to each class, however. $\endgroup$
    – Dave
    Commented Dec 13, 2020 at 4:10
  • $\begingroup$ And since you know the class to which each prediction belongs, you color-code the graph. $\endgroup$
    – Dave
    Commented Dec 13, 2020 at 4:18
  • $\begingroup$ Dear Dave, the 0.8 is misclassified because the model predicted a benign patient as a malignant patient - this should contribute the false positive right? How then would the graph tell us that this 1 single patient is under the false positive area? $\endgroup$
    – nan
    Commented Dec 13, 2020 at 4:21
  • $\begingroup$ It’s a green patient with a predicted probability over the threshold. That’s how it’s a false positive. $\endgroup$
    – Dave
    Commented Dec 13, 2020 at 4:24
  • $\begingroup$ Thanks Dave for answering me patiently. I do have some solid background on deep learning in general. But did not really look into the metric details like ROC-AUC. It kinda of threw me off with the graph above. So the color coding is deliberate to indicate our ground truth in a way. Right? $\endgroup$
    – nan
    Commented Dec 13, 2020 at 4:37

1 Answer 1

  1. Those are the probabilities predicted by the model. If you have a million patients in each group, you plot the million probabilities predicted for the red patients and the million probabilities predicted for the green patients.

  2. To evaluate any model, we must compare to the true values. The patients in green are the ones who do not have cancer, and the patients in red do have cancer.

  • $\begingroup$ Thanks for the reply. I have some comments to ask. And I edited in the question as it’s too long. $\endgroup$
    – nan
    Commented Dec 13, 2020 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.