Help with understanding statistical measures and Receiver Operating Characteristics

Question

In my machine learning class we just went over statistical measures and plots. We looked at the definitions of True Positive Rate (sensitivity/recall, etc), 1-False Positive Rate (speciicity), Accuracy, and Precision. Defining them as the following

True Positive rate (Sensitivity/Recall) = True Positives/(True Positives + False Negatives)

1 - False Positive Rate (specificity) = True Negatives/(True Positives + False NegativeS)

Accuracy = (True Positive + True Negative)/(Positives + Negatives)

Precision = True Positive/(True Postitive + False Negative)

When my professor asked what exactly the True Positive Rate measured many people said "Its the percentage of correctly classified samples). And to each person who replied with this kind of answer he said "sorta" but he never really explained exactly what it was. What does each of these exactly measure and how does it help us examine data? Is there a way one of these is useful for a quantitative measure of uncertainty?

We also went over ROC plots and Precision-Recall Plots.

He said that since for each run of data we can only get one value for each of these measures we create these plots of True Positive Rate vs False negative Rate and Precision vs Recall by adjusting some sort of threshold (number of true positives??) and then plotting a point when we get to that threshold. What exactly do we adjust the threshold of to generate these two types of plots.

As i understand it the plot on the ROC curve with the most area is the best at predicting a sample, is this also true for the precision-recall plots?

Answer 1

Consider the case you have a metal detector, and you want to search for treasures in the seashore. Unfortunately your metal detector it's not very accurate, and sometimes it sounds even if there's no metal, but just a stupid rock. Since the device emits different sound tones, you want to find the best tone to start digging, and therefore maximize the profits. Thus, for every tone level threshold you have 4 cases when the metal detector sounds:

1. The tone exceed the threshold, and there is really a treasure [True Positive] :)
2. The tone exceed the threshold, but there is not a treasure [False Positive] :(
3. The tone do not exceed the threshold, but there is a treasure [False Negative] :(
4. The tone do not exceed the threshold, and there is not a treasure [True Negative] :)

Obviously you would like that cases 1 and 4 happen most, since it would mean that you dig only when there's a treasure, and you would waste no time and energy and you would be rich soon.

Your problem is thus to find what's the right tone that alerts you to dig only when there is really a treasure.

We can define the following metrics:

$Accuracy = \frac{TP+TN}{TP+TN+FP+FN}$ Define how accurately we classify the data.

$Precision =\frac{TP}{TP+FP}$ Precision defines how often what you predict as true it is really true. Back to our example, low precision means you will dig quite always and do not miss any treasure, with high precision you will dig barely but you would reasonably miss some treasures. For medical application you may prefer to make some more tests and do not miss a hit rather than assure your patient is healthy when she/he is not.

$Sensitivity =\frac{TP}{TP+FN}$ Sensitivity defines the portion of true elements classified as 'true'. In other words is the hit rate. Also called True Positive Rate.

$Specificity =\frac{TN}{FP+TN}$ It is the mirror of the sensitivity for the opposite class.

$False Positive Rate = \frac{FP}{FP+TN}$ In contrast with the TPR (precision) it measures the number of false alarm over negatives.

Also other metrics based on TP, FP, TN, FN exist.

Curve

You can easily see that threshold, TPR and FPR are highly related, in fact if you use as threshold more than the highest tone the device can emit, you would never dig to find a treasure, since the device would never sound. With such threshold you will have TPR and FPR to zero, since you will have no positive. Moving the threshold to the lowest tone, the treasures' hunter registers for each threshold the TPR and FPR rate. When threshold is set to the lowest tone he will have no negatives and will dig everywhere, and thus TPR and FPR will be one (if normalized).

At the end of the process he will have a ROC curve where for each scatter he knows its associated threshold. Axes are normalized from 0 to 1. Commonly used x-axis represents FPR rate and y-axis the TPR rate. The worst case can happen is that the resulting plot is a straight line from [0,0] to [1,1]; this would mean that you classify for each threshold with probability $P(x) = 0.5$. This would mean that using a coin to predict whether there is rock or a metal would have the same effect. If instead the resulting plot is a curve you can now find the desired threshold. A common and mathematically proved choice, is to select the threshold linked to the intersection point between the curve and the straight line passing through the points [0,1] [1,0], since this point would maximize the ration $\frac{TPR}{FPR}$. Even if this seem to be the most desired choice, it is important to recall that for different kinds of problems you may wish to give more importance to a metric rather to another. It is reasonably to suppose that for a medical application to detect cancer, you may give less importance to have a high FPR, since it is better to have a false alarm rather to miss a cancer.

Finally I want to give you the last information about ROC. In fact, they are also a tool for comparing different classification algorithm. Let's return to the starting example, you can see the metal detector as a classification algorithm. The treasures' hunter now want to find which is the best metal detector out on the market. So he buy all the devices available and for each of them he makes the same tests and obtains a different ROC for each device. A reasonable metric to compare the metal detectors is to measure the area under the ROC curve (AUC). The perfect metal detector would sum its values to 1 ($ AUC = 1$), since it would classify always correctly. Usually algorithms with AUC values over 0.7 are considered good.

Answer 2

To directly address in your professor's "sorta":

The TP rate you describe is not the the fraction of correctly classified samples. It's the fraction of correctly classified positive samples.

The correctly classified fraction of all samples is what you defined as accuracy.