3
$\begingroup$

What is meant by cutoff value or threshold value ? how will you explain the ROC curve with respect to fpr and tpr.ROC example

$\endgroup$
3
  • $\begingroup$ Can you explain us what is fpr, tpr? $\endgroup$ Commented Sep 22, 2016 at 12:19
  • $\begingroup$ FPR stands for False positive rate and it means " Incorrectly Identified" whereas TPR stands for True Positive Rate which means "Correctly Identified" $\endgroup$
    – newbie
    Commented Sep 22, 2016 at 15:17
  • $\begingroup$ My answer to a digital signal processing question addresses precision-recall curves, but the idea is the same. $\endgroup$
    – Dave
    Commented Jan 25 at 15:02

2 Answers 2

1
$\begingroup$

Motivation

Suppose we have a (machine learning) model trying to classify a data point between two classes, the model assigns to a data point a score (usually this score is a number between 0 and 1). The score is a measure of how confident the model is that the data point belongs to a certain class (sometimes the score is a calibrated probability, although not always).

For example, suppose you are trying to classify whether an email is spam or not spam.

The model scores the first email as 0.9 and the next email as 0.4. The model is in some sense more confident that the first email is spam than the second.

Having the raw scores is good, but sometimes in the real world we need to make a hard classification so we can take action off it, in this example will we block the email.

We can set a threshold (or cutoff), and say that any score above this value will be marked as one class, and below it as the other. For example we might say anything above 0.7 we will mark as "spam", and anything below it as "not spam". Please take note that the threshold does not have to be 0.5, even though this is often the default value in some popular packages.

Definitions

When we set our threshold and classify anything above it as spam and below it as not spam, we will (probably) get some classifications correct and some incorrect. There are different things that can happen, we will call spam emails the positive class:

  • We correctly mark a spam email as spam (TP - True Positive)
  • We correctly mark a not spam email as not spam (TN - True Negative)
  • We incorrectly mark a not spam email as spam (FP - False Positive)
  • We incorrectly mark a spam email as not spam (FN - False Negative)

Using these we can define the following two quantities:

True Positive Rate (Also known as sensitivity or recall)

$$\text{TPR} = \frac{\text{TP}}{\text{TP} + \text{FN}}$$

False Positive Rate (Also known as fall-out)

$$\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}$$

You could set your threshold in different places, doing this would give you different values for TP, TN, FP, FN and consequently different values for the TPR - true positive rate and FPR - false positive rate.

For example if you set the threshold at 0 and so marked every email as spam, then you would have correctly marked every positive as a positive, so there are no false negatives, and so the TPR = 1. On the other hand if you set the threshold at 1 and so marked every email as not spam, then you would have no false positives, and FPR=0.

So depending where you set your threshold, you will have different values for the true positive rate and false positive rate.

Now, imagine if you try every possible threshold, and for each threshold value you calculate the false positive rate and true positive rate, and plot this curve with the x-axis as the false positive rate and the y-axis as the true positive rate - this is the ROC - curve which you have in the diagram in the original post.

Why is it useful?

This was not part of the original question, but it is worth briefly mentioning. The area under the ROC curve, (often ROC-AUC or AUCROC) tells us something about how 'good' the classifier is - in particular it tells us something about the discriminative ability of the classifier.

  • A ROC-AUC of 1 means the model has perfect discriminative ability.

  • A ROC-AUC of 0.5 means the model is performing the same as random guessing between the two classes. The performance of this classifier would be a diagonal line from bottom left to top right of the ROC space, you can see this on the diagram in the original post.

  • A ROC-AUC below 0.5 indicates the model is performing worse than random chance, consistently making incorrect predictions. In this case it may be worth checking if the class labels are being flipped.

There is a probabilistic interpretation for the ROC-AUC, if you randomly select an observation from both classes (spam - positive class, not spam - negative class), then the ROC-AUC is the probability that the model scores the positive observation higher than the score for the negative observation.

Given thresholds were mentioned in the original post, it is worth noting that the ROC-AUC gives us a measure of the discriminative ability of a model which is independent of the threshold that is chosen.

Further Reading

$\endgroup$
1
  • $\begingroup$ I once asked about spam vs ham emails in particular, and I got an interesting response from Stephan Kolassa. $//$ Please take note that the threshold does not have to be 0.5, even though this is often the default value in some popular packages. There are advantages to looking at the raw outputs, but I definitely feel that many machine learning misconceptions come from the idea that $0.5$ is a magic threshold that must be used, since that is the default in a popular software package. $\endgroup$
    – Dave
    Commented Jan 25 at 15:04
1
$\begingroup$

Suppose we have a human being who's trying to decide whether some measurement corresponds to a disease state or a healthy sample. He has a set of samples from people known to have the disease and another set from healthy people ("controls"). Measuring each of these sets produces two distributions of serum concentrations for a particular biomarker that may be helpful in diagnosing the disease. But these two distributions overlap somewhat, so they are not perfectly able to tell the difference between a disease sample and a healthy sample.

Why would this be so?

Because diseases can result from many different combinations of events. Some healthy people may high high levels of a particular marker and yet never develop an associated disease due to other factors being present. He decides to draw a line somewhere along the two distributions and classify samples higher than the line diseased and samples below the line healthy. This results in some number of correctly diagnosed disease samples and some number of incorrect false positives. Each point on the ROC curve corresponds to one of these threshold lines. By moving the line up or down the whole curve can be traced out.

If you're very lucky you might find a biomarker that yields a 100% true positive and 0% false positive rate for a particular threshold value--the distributions have no overlap, so all you need to do is draw the line in the right place. By lowering the threshold, you start to get false positives but the true positives stay at 100%. Such a "curve" would look like a square, rising instantly to 100% true positives and then moving to the right as false positives increased. The area under the curve (AUC) would be 1. In reality there may never be such a perfect case but the closer you can get to an AUC of 1, the better your marker is at differentiating between a healthy or a diseased sample.

Most curves never get to 100% true positive before moving along the x-axis, so they look like going up a hill to a plateau. In these cases the threshold that produces the largest possible true positive rate comes at the cost of some number of false positives. Some markers can even have predictive power so they can forecast a disease occurring years before any symptoms are experienced.

$\endgroup$
1
  • $\begingroup$ Welcome to Cross Validated! I've upvoted but also partitioned your answer into paragraphs so it's a bit easier on the eyes, but the content is all the same. Feel free to edit the paragraphs as you deem appropriate. $\endgroup$
    – Dave
    Commented Jan 25 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.