5
$\begingroup$

If I have the following sensitivity and specificity values, what is the best decision we can say in this case?

sensitivity     specificity
-----------     -----------
  66.3             74.7
  87.2             65.9
  56.4             76.4
  79.5             94.3

Thanks.

Improve this question
$\endgroup$
1
  • 10
    $\begingroup$ The answer will depend on your problem at hand: are both types of error equally bad, or are (e.g.) false negatives way worse than false positives. For instance if you want to test for a disease that kills, you want to strongly avoid false negatives (because those people will die when you don't pick up the presence of the disease with your test) $\endgroup$
    – Nick Sabbe
    Aug 11, 2011 at 13:54

2 Answers 2

Reset to default
11
$\begingroup$

To make an optimal decision you need to know all relevant data about an individual (used to estimate the probability of an outcome), and the utility (cost, loss function) of making each decision. Sensitivity and specificity do not provide this information. That's why direct probability models such as the binary logistic model are so popular. For example, if you estimated that the probability of a disease given age, sex, and symptoms is 0.1 and the "cost" of a false positive equaled the "cost" of a false negative, you would act as if the person does not have the disease. Given other utilities you might make different decisions. If the utilities are unknown, you give the best estimate of the probability of the outcome to the decision maker and let her incorporate her own unspoken utilities in making an optimum decision for her.

Besides the fact that cutoffs do not apply to individuals, only to groups, individual decision making does not utilize sensitivity and specificity. For an individual we can compute $\textrm{Prob}(Y=1 | X=x)$; we don't care about $\textrm{Prob}(Y=1 | X>c)$, and an individual having $X=x$ would be quite puzzled if you gave her $\textrm{Prob}(X>c | \textrm{future unknown Y})$ when they already know $X=x$ so it is no longer a random variable.

Even when group decision making is needed, sensitivity and specificity can be bypassed. For mass marketing, for example, you can rank order individuals by the estimated probability of buying the product, to create a lift curve. This is then used to target the $k$ most likely buyers where $k$ is chosen to meet total program cost constraints.

Improve this answer
$\endgroup$
2
  • $\begingroup$ @DrHarrell. I understand you pt. about no cutoffs for individuals. However, I don't understand the next sentences in the paragraph. Could you pls elaborate? $\endgroup$
    – suncoolsu
    Aug 12, 2011 at 14:03
  • 1
    $\begingroup$ I'm not sure I understand what you need. If referring to the second paragraph, let $X$ = the single predictor or the predicted probability that $Y=1|X$ and $c$ be a cutoff (naively) chosen to trigger a "positive" decision about $Y$, where $Y=1$ indicates that the event occurs and $Y=0$ that it did not. Sensitivity is Prob$(X>c | Y=1)$ which is in backwards time (and thus information flow) order. What is needed is Prob$(Y=1 | X=x)$ where $x$ is the observed value for the object to be decided upon. Once we know $X=x$ it is not helpful to know that $X>c$ for some cutoff $c$. $\endgroup$
    – Frank Harrell
    Aug 12, 2011 at 21:18
5
$\begingroup$

enter image description here

As Nick has already pointed out the answer depends on context. However, if you only had to judge based on the values of sensitivity and specificity values that you provided, then a good strategy is to plot the sensitivity on the y-axis and $(100\%-$ specificity$)$ on the x-axis and look for the highest leftmost point. This point would be the sensitivity and specificity pair that you might want to choose. But please remember that domain knowledge plays a huge impact on the final decision. Here is the R-code that does the work for you:

R> sens <- c(66.3, 87.2, 56.4, 79.5)
R> spec <- c(74.7, 65.9, 76.4, 94.3)
R> df <- data.frame(y=sens, x=(100-spec))
R> df
     y    x
1 66.3 25.3
2 87.2 34.1
3 56.4 23.6
4 79.5  5.7
R> df <- df[order(df$x), ]
R> df
     y    x
4 79.5  5.7
3 56.4 23.6
1 66.3 25.3
2 87.2 34.1  

R> plot(x = df$x, y = df$y, type = "b", 
     pch = 20, lty="solid", lwd = 2, 
     main = "Sensitiviy and (100-Specificity) Curve", 
     xlab = "100 - Specificity", ylab = "Sensitivity")

Therefore, you'd choose 79.5, 94.3 sensitivity and specificity pair.

Basic Idea: High number for true positives and low number for false positives!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.