2
$\begingroup$

Consider the model

fit2 <- glm(y~x+z,data=records,family=binomial)

I have about 42000 records, of which close to 38000 belong to class y=0 and the remaining 4000 belong to class y=1. In order for me to compute the confusion matrix, I need to select a threshold t against which I need to compare the probabilities of the above model. How do I select this t?

Assuming my positive class to be the rare class, the risk of predicting a positive instance as negative is higher than the risk of predicting a negative instance as positive.

Should I use the threshold that maximizes sensitivity alone? or should I use the threshold that maximizes both sensitivity and specificity?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
5
$\begingroup$

Maximizing only sensitivity is trivial: Take an extreme threashold such that all subjects are considered positive. Then of course specificity is poor. So you take both, sensitivity and specificity simultaneously.

One criterion is the Youden index: The sum of sensitivity and specificity has to be maximal. However, you can choose a different risk function weighting the trade-off between sensitivity and specificity according to your practical insight into the field of application. For example, if you want to diagnose a severe disease, you would prefer higher sensitivity over high specificity. But if the treatment for this disease has also side effects, you would not want to expose so many patients to the treatment by tolerating low specificity.

Improve this answer
$\endgroup$
2
  • $\begingroup$ my question is -- how do you weight the trade-off between sensitivity and specificity? When you try to maximize both jointly or as a sum, in either case, you end up with just one value of threshold and its corresponding sensitivity and specificity. $\endgroup$
    – rk567
    Jan 14, 2015 at 19:58
  • $\begingroup$ OK, what I answered applys only to one-dimensional predictors. But you have two of them in your model, x and y. So your question is rather how to find some line of thresholds dividing the 2-dimensional plane spanned by x and y in two parts, right? $\endgroup$
    – Horst Grünbusch
    Jan 15, 2015 at 15:45
0
$\begingroup$

What do you mean by setting a treshold? Which treshold? Do you mean playing with the setting of a probability like if >0.5 y = 1 or if >0.4 y = 1 to divide them over the two different classes to play with the hight of sensitivity and specificity?

I agree that it depends on the importance. Once it is related to a disease you can use the comment above, once it is business related you can use the principles of an expected value framework. This resource explains this:

http://people.stern.nyu.edu/padamopo/blog/DataScienceTeaching/Lecture%206%20-%20Decision%20Analytic%20Thinking.pdf

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.