1
$\begingroup$

Let's say I am doing logistic regression.

I split my data into training and test.

I get an ROC for my training data and it has a cut-off of 0.25

I calculate my evaluation metrics, let's say just specificity and get 70%.

Then I do an ROC for my test data, and it has a cut-off of 0.4

If I use the cutoff value from training ROC (of 0.25) I get a specificity of 55%, but if I use cutoff from test ROC (of 0.4) I get specificity of 68%.

Which one do I use and why?

Improve this question
$\endgroup$
1
  • $\begingroup$ If your goal is to maximize the test scores then use the test ROC cutoff. $\endgroup$
    – user2974951
    Commented Dec 12, 2018 at 7:34

2 Answers 2

Reset to default
1
$\begingroup$

This can be viewed as parameter estimation and it's usually done on a dedicated validation set, i.e. you split your data into three sets A, B and C then train the model on A, choose an appropriate cut-off using predictions on B and estimate the final generalization on C.

Improve this answer
$\endgroup$
1
$\begingroup$

Please tell us from what source you learned this procedure. It is entirely wrong and represents bad statistical practice. Here are some of the things that are going wrong:

  • Logistic regression is used to estimate the probability of an event, not for classification
  • Data splitting as a validation method (whether using 2 or 3 subsets) only works when you start with an enormous sample size, otherwise it is volatile
  • ROC curves are not compatible with decision making - see the Diagnosis chapter of BBR
  • ROC curves should not be used for obtaining a cutoff value - see the Information Loss chapter of BBR as well as http://www.fharrell.com/tags/classification and http://www.fharrell.com/post/mlconfusion/

Optimal decisions are made by taking the risk estimate from logistic regression and combining it with the utiity/loss/cost function, to maximize expected utility. If you dichotomize the risk estimate before applying it to the utility function, the resulting decision will not be optimal. Except for that, everything else I've mentioned here is covered in my Regression Modeling Strategies book and course notes.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Yes, logistic regression is used for estimating probabilities but IMO saying that it's not used for classification is overly pedantic, as it is very often used with a classification context in mind. Also, I don't see a reason for why ROC curves shouldn't be used for deciding on a decision threshold. The part in the chapter why supposedly addresses is merely 5 lines long and basically just states ROC curves are bad for decision making, without bothering to explain why. $\endgroup$
    – Scholar
    Commented Jan 17, 2019 at 14:15
  • $\begingroup$ IMO ROC curves can be useful for decision making, for example if I want to identify cases in a set of cases and controls and want to limit the fraction of false-discoveries to say 0.05, I can do this by selecting the corresponding threshold on a ROC curve. $\endgroup$
    – Scholar
    Commented Jan 17, 2019 at 14:15
  • $\begingroup$ I actually learnt this in final year undergrad at uni, the stats course was quite a mess. We uses ROC for decision making for binary classification using logistic regression. $\endgroup$
    – Alexis Drakopoulos
    Commented Jan 17, 2019 at 16:05
  • $\begingroup$ If you looked at the references I provided you'd see the "why". The central problem with ROC curves for decision making is that they have transposed conditionals as discussed so nicely by Drew Levy in his "Matrix of Confusion" article I reference. Decisions need to be made on the basis of P(outcome) not P(input). And note that any classification you do after doing logistic regression needs to be removed from the logistic regression context, and is usually unnecessary. $\endgroup$
    – Frank Harrell
    Commented Jan 17, 2019 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.