Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Usually in logistic regression, we fit a model and get some predictions on the training set. We then cross-validate on those training predictions (something like here) and decide the optimal threshold value based on something like the ROC curve.
Why don't we incorporate cross-validation of the threshold INTO the actual model, and train the whole thing end-to-end?
A threshold isn't trained with the model because logistic regression isn't a classifier (cf., Why isn't Logistic Regression called Logistic Classification?). It is a model to estimate the parameter, $p$, that governs the behavior of the Bernoulli distribution. That is, you are assuming that the response distribution, conditional on the covariates, is Bernoulli, and so you want to estimate how the parameter that controls that variable changes as a function of the covariates. It is a direct probability model only. Of course, it can be used as a classifier subsequently, and sometimes is in certain contexts, but it is still a probability model.
It's because the optimal threshold is not only a function of the true positive rate (TPR), the false positive rate (FPR), accuracy or whatever else. The other crucial ingredient is the cost and the payoff of correct and wrong decisions.
If your target is a common cold, your response to a positive test is to prescribe two aspirin, and the cost of a true untreated positive is an unnecessary two days' worth of headaches, then your optimal decision (not classification!) threshold is quite different than if your target is some life-threatening disease, and your decision is (a) some comparatively simple procedure like an appendectomy, or (b) a major intervention like months of chemotherapy! And note that although your target variable may be binary (sick/healthy), your decisions may have more values (send home with two aspirin/run more tests/admit to hospital and watch/operate immediately).
Bottom line: if you know your cost structure and all the different decisions, you can certainly train a decision support system (DSS) directly, which includes a probabilistic classification or prediction. I would, however, strongly argue that discretizing predictions or classifications via thresholds is not the right way to go about this.
See also my answer to the earlier "Classification probability threshold" thread. Or this answer of mine. Or that one.
Philosophical concerns aside, this would cause computational difficulties.
The reason why is that functions with continuous output are relatively easy to optimize. You look for the direction where the function increases, and then go that way. If we alter our loss function to include the "cutoff" step, our output becomes discrete, and our loss function is therefore also discrete. Now when we alter the parameters of our logistic function by "a little bit" and jointly alter the cutoff value by "a little bit", our loss gives an identical value, and optimization becomes difficult. Of course, it's not impossible (There's a whole field of study in discrete optimization) but continuous optimization is by far the easier problem to solve when you are optimizing many parameters. Conveniently, once the logistic model has been fit, finding the optimal cutoff, though still a discrete output problem, is now only in one variable, and we can just do a grid search, or some such, which is totally viable in one variable.
Regardless of the underlying model, we can work out the sampling distributions of TPR and FPR at a threshold. This implies that we can characterize the variability in TPR and FPR at some threshold, and we can back into a desired error rate trade-off.
A ROC curve is a little bit deceptive because the only thing that you control is the threshold, however the plot displays TPR and FPR, which are functions of the threshold. Moreover, the TPR and FPR are both statistics, so they are subject to the vagaries of random sampling. This implies that if you were to repeat the procedure (say by cross-validation), you could come up with a different FPR and TPR at some specific threshold value.
However, if we can estimate the variability in the TPR and FPR, then repeating the ROC procedure is not necessary. We just pick a threshold such that the endpoints of a confidence interval (with some width) are acceptable. That is, pick the model so that the FPR is plausibly below some researcher-specified maximum, and/or the TPR is plausibly above some researcher-specified minimum. If your model can't attain your targets, you'll have to build a better model.
Of course, what TPR and FPR values are tolerable in your usage will be context-dependent.
For more information, see ROC Curves for Continuous Data by Wojtek J. Krzanowski and David J. Hand.
Usually in biomedical research, we don't use a training set---we just apply logistic regression on the full dataset to see which predictors are significant risk factors for the outcome we're looking at; or to look at one predictor of interest while controlling for the effect of other possible predictors on the outcome.
I'm not sure quite what you mean by threshold values, but there are various parameters that one may seek to optimize: AUC, cutoff values for a dichotomizing a continuous predictor variable, positive and negative predictive values, confidence intervals and p-values, false positive and false negative rates.
Logistic regression looks at a population of subjects and assesses the strength and causal direction of risk factors that contribute to the outcome of interest in that population. It's also possible to "run it in reverse," so to speak, and determine an individual's risk of the outcome given the risk factors that the individual has. Logistic regression assigns each individual a risk of the outcome, based on their individual risk factors, and by default this is 0.5. If a subject's probability of having the outcome (based on all the data and subjects in your model) is 0.5 or above, it predicts he will have the outcome; if below 0.5 then it predicts he won't. But you can adjust this cutoff level, for example to flag more individuals who might be at risk of having the outcome, albeit at the price of having more false positives being predicted by the model. You can adjust this cutoff level to optimize screening decisions in order to predict which individuals would be advised to have further medical followup, for example; and to construct your positive predictive value, negative predictive value, and false negative and false positive rates for a screening test based on the logistic regression model. You can develop the model on half your dataset and test it on the other half, but you don't really have to (and doing so will cut your 'training' data in half and thus reduce the power to find significant predictors in the model). So yes, you can 'train the whole thing end to end'. Of course, in biomedical research, you would want to validate it on another population, another data set before saying your results can be generalized to a wider population. Another approach is to use a bootstrapping-type approach where you run your model on a subsample of your study population, then replace those subjects back into the pool and repeat with another sample, many times (typically 1000 times). If you get significant results a prescribed majority of the time (e.g. 95% of the time) then your model can be deemed validated---at least on your own data. But again, the smaller the study population you run your model on, the less likely it will be that some predictors will be statistically significant risk factors for the outcome. This is especially true for biomedical studies with limited numbers of participants.
Using half of your data to 'train' your model and then 'validating' it on the other half is an unnecessary burden. You don't do that for t-tests or linear regression, so why do it in logistic regression? The most it will do is let you say 'yeah it works' but if you use your full dataset then you determine that anyway. Breaking your data into smaller datasets runs the risk of not detecting significant risk factors in the study population (OR the validation population) when they are in fact present, due to small sample size, having too many predictors for your study size, and the possibility that your 'validation sample' will show no associations just from chance. The logic behind the 'train then validate' approach seems to be that if the risk factors you identify as significant aren't strong enough, then they won't be statistically significant when modeled on some randomly-chosen half of your data. But that randomly-chosen sample might happen to show no association just by chance, or because it is too small for the risk factor(s) to be statistically significant. But it's the magnitude of the risk factor(s) AND their statistical significance which determine their importance and for that reason it's best to use your full dataset to build your model with. Statistical significance will become less significant with smaller sample sizes, as it does with most statistical tests.
Doing logistic regression is an art almost as much as a statistical science. There are different approaches to use and different parameters to optimize depending on your study design.
