# Philosophical question on logistic regression: why isn't the optimal threshold value trained?

Usually in logistic regression, we fit a model and get some predictions on the training set. We then cross-validate on those training predictions (something like here) and decide the optimal threshold value based on something like the ROC curve.

Why don't we incorporate cross-validation of the threshold INTO the actual model, and train the whole thing end-to-end?

A threshold isn't trained with the model because logistic regression isn't a classifier (cf., Why isn't Logistic Regression called Logistic Classification?). It is a model to estimate the parameter, $$p$$, that governs the behavior of the Bernoulli distribution. That is, you are assuming that the response distribution, conditional on the covariates, is Bernoulli, and so you want to estimate how the parameter that controls that variable changes as a function of the covariates. It is a direct probability model only. Of course, it can be used as a classifier subsequently, and sometimes is in certain contexts, but it is still a probability model.

• Okay, I understand that part of the theory (thank you for that eloquent explanation!) but why can't we incorporate the classification aspect into the model? That is, why can't we find p, then find the threshold, and train the whole thing end-to-end to minimize some loss? – StatsSorceress Apr 25 at 15:55
• You certainly could (@Sycorax's answer speaks to that possibility). But because that isn't what LR itself is, but rather some ad hoc augmentation, you would need to code up the full optimization scheme yourself. Note BTW, that Frank Harrell has pointed out that process will lead to what might be considered an inferior model by many standards. – gung Apr 25 at 16:02
• Hmm. I read the accepted answer in the related question here, and I agree with it in theory, but sometimes in machine learning classification applications we don't care about the relative error types, we just care about "correct classification". In that case, could you train end-to-end as I describe? – StatsSorceress Apr 25 at 16:29
• As I said, you very much can set up your own custom optimization that will train the model & select the threshold simultaneously. You just have to do it yourself & the final model is likely to be poorer by most standards. – gung Apr 25 at 18:13
• @StatsSorceress "... sometimes in machine learning classification ...". There should be a big emphasis on sometimes. It's hard to imagine a project where accuracy is the correct answer. In my experience, it always involves precision and recall of a minority class. – Wayne Apr 26 at 12:10

It's because the optimal threshold is not only a function of the true positive rate (TPR), the false positive rate (FPR), accuracy or whatever else. The other crucial ingredient is the cost and the payoff of correct and wrong decisions.

If your target is a common cold, your response to a positive test is to prescribe two aspirin, and the cost of a true untreated positive is an unnecessary two days' worth of headaches, then your optimal decision (not classification!) threshold is quite different than if your target is some life-threatening disease, and your decision is (a) some comparatively simple procedure like an appendectomy, or (b) a major intervention like months of chemotherapy! And note that although your target variable may be binary (sick/healthy), your decisions may have more values (send home with two aspirin/run more tests/admit to hospital and watch/operate immediately).

Bottom line: if you know your cost structure and all the different decisions, you can certainly train a decision support system (DSS) directly, which includes a probabilistic classification or prediction. I would, however, strongly argue that discretizing predictions or classifications via thresholds is not the right way to go about this.

Philosophical concerns aside, this would cause computational difficulties.

The reason why is that functions with continuous output are relatively easy to optimize. You look for the direction where the function increases, and then go that way. If we alter our loss function to include the "cutoff" step, our output becomes discrete, and our loss function is therefore also discrete. Now when we alter the parameters of our logistic function by "a little bit" and jointly alter the cutoff value by "a little bit", our loss gives an identical value, and optimization becomes difficult. Of course, it's not impossible (There's a whole field of study in discrete optimization) but continuous optimization is by far the easier problem to solve when you are optimizing many parameters. Conveniently, once the logistic model has been fit, finding the optimal cutoff, though still a discrete output problem, is now only in one variable, and we can just do a grid search, or some such, which is totally viable in one variable.

Regardless of the underlying model, we can work out the sampling distributions of TPR and FPR at a threshold. This implies that we can characterize the variability in TPR and FPR at some threshold, and we can back into a desired error rate trade-off.

A ROC curve is a little bit deceptive because the only thing that you control is the threshold, however the plot displays TPR and FPR, which are functions of the threshold. Moreover, the TPR and FPR are both statistics, so they are subject to the vagaries of random sampling. This implies that if you were to repeat the procedure (say by cross-validation), you could come up with a different FPR and TPR at some specific threshold value.

However, if we can estimate the variability in the TPR and FPR, then repeating the ROC procedure is not necessary. We just pick a threshold such that the endpoints of a confidence interval (with some width) are acceptable. That is, pick the model so that the FPR is plausibly below some researcher-specified maximum, and/or the TPR is plausibly above some researcher-specified minimum. If your model can't attain your targets, you'll have to build a better model.

Of course, what TPR and FPR values are tolerable in your usage will be context-dependent.

For more information, see ROC Curves for Continuous Data by Wojtek J. Krzanowski and David J. Hand.

• This doesn't really answer my question, but it's a very nice description of ROC curves. – StatsSorceress Apr 25 at 15:51
• In what way does this not answer your question? What is your question, if not asking about how to choose a threshold for classification? – Sycorax Apr 25 at 15:52
• I'm not aware of any statistical procedure that works that way. Why is this square wheel a good idea? What problem does it solve? – Sycorax Apr 25 at 15:56
• "How do I choose a threshold in a way that reduces training time?" seems like a very different question from the one in your original post. – Sycorax Apr 25 at 16:29
• Regardless, I don't see how this saves time. Making an ROC curve is not the most expensive part of estimating a model, so moving threshold choice into the optimization step seems ad hoc and unnecessary. – Sycorax Apr 25 at 16:32

Usually in biomedical research, we don't use a training set---we just apply logistic regression on the full dataset to see which predictors are significant risk factors for the outcome we're looking at; or to look at one predictor of interest while controlling for the effect of other possible predictors on the outcome.