3
$\begingroup$

I created a probit model and tested it against a random sub sample of my dataset. I am interested specifically in seeing how many data points I can predict to be FALSE without having too many that are actually TRUE. Using the threshold of 0.1 (see below), I was able to predict to be FALSE about 30% with a false negative rate of 2.5%.
However, I don't know if this is optimal.

Is there a way for me to pick my threshold that maximizes my FALSE predictions while minimizing my false negatives?

glm.fit=glm(Outcome~A+B+C+D+E+F+G,data=myData,family=binomial(link="probit"))
test=mysample <- myData[sample(1:nrow(myData),10000,replace=FALSE),]
glm.probs =predict(glm.fit,test, type="response")
glm.pred=rep(0,10000)
glm.pred[glm.probs>.1]=1
x <- sum(glm.pred == 0 & test$Outcome == 1)
y <- sum(glm.pred == 0) 
$\endgroup$
  • $\begingroup$ I'm guess you mean a $threshold$ that maximizes your True negative rate while minimizing your false negative rate? $\endgroup$ – Andrew Cassidy Mar 7 '14 at 17:30
  • $\begingroup$ I believe this has been covered here before. You're choosing a point on the AUC curve. To do so you need a cost/utility function that balances False Negatives/False Positives. Without the cost function you can't optimize anything. $\endgroup$ – charles Mar 7 '14 at 17:30
  • $\begingroup$ If I predict it will be TRUE, then I have to do x amount of work. If I predict it will be FALSE, then I don't have to do any work. So I want to do as little work as possible but if I don't do the work, there is a larger cost associated with that, and obviously a smaller cost associated with doing that work up front. $\endgroup$ – zu2122 Mar 7 '14 at 17:32
  • $\begingroup$ @charles you mean ROC curve not AUC curve $\endgroup$ – Andrew Cassidy Mar 7 '14 at 17:33
  • $\begingroup$ @AndrewCassidy yes! $\endgroup$ – charles Mar 7 '14 at 17:46
0
$\begingroup$

This threshold is always a trade off. Take a look at Reciever Operating Charateristic (ROC cruves). Try the R library ROCR too.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.