0
$\begingroup$

I have a very simple (medical) data set with one continues independent variable X(a biomarker measurement) and y - the dependent outcome variable (binary, disease/non-disease). The question is to find the optimal cutpoint to separate two groups.

My first instinct was to use a logistic regression with 10-fold cross validation and computed metrics of interest for each fold (AUC, PPV, NPV, Se, Sp, LR+, LR-). But after doing some reading, I realised that I can use directly the independent variable X as the ranking variable (this is what exactly logistic regression does, maps input variable to ranking probabilities, please correct me if I'm wrong). Obviously I've got slightly different results and thus two questions:

QUESTION 1: How to extract the cutpoint from k-fold cross-validated logistic regression?

AFAIK, cross validation is used to roughly estimate the generalisation error and fine tune the hyperparameters (if there are any). Every cross validation fold produces a surrogate model. Once the hyper parameters are tuned, then all data used to fit the model. From this model, one can extract the cutpoint (inverse logistic transformation).

QUESTION 2: Is there a conceptual difference between direct use of the independent variable and mapping through cross-validated logistic regression?

$\endgroup$
1
$\begingroup$

QUESTION 1: How to extract the cutpoint from k-fold cross-validated logistic regression?

AFAIK, cross validation is used to roughly estimate the generalisation error and fine tune the hyperparameters (if there are any). Every cross validation fold produces a surrogate model. Once the hyper parameters are tuned, then all data used to fit the model. From this model, one can extract the cutpoint (inverse logistic transformation).

Small but important note on the procedure: you may use cross-valdiation to estimate generalization error xor for hyperparameter tuning. I.e., once you've used the CV results for hyperparameter tuning, the respective test cases have become part of training the final model and are not independent any more and cannot be used to estimate generalization error. Look up nested cross validation for more information.

Now, the cutpoint is just a hyperparameter, so it is determined during hyperparameter tuning. E.g. in the inner loop of a nested CV setup.

In other words, from the point of view of the outer cross validation (the one used to estimate generalization error), you determine a number of surrogate models with possibly varying cutpoints.
Checking the stability of the auto-tuned cutpoints across the outer surrogate models should be one point of your validation (or rather, verification) procedure.

The final model is trained on all data, and this training again performs the inner CV auto-tuning of the cutpoint.

QUESTION 2: Is there a conceptual difference between direct use of the independent variable and mapping through cross-validated logistic regression?

There is no conceptual difference between cross validation auto-tuning of the cutpoint directly on x and cross validation auto-tuning of the cutpoint on a logistic regression on x.

Logistic regression predicts posterior probability, though. Thus, you can check whether it is well calibrated, i.e. whether the claimed posterior probabilities are reasonably close to the observed probabilities.
You cannot do a similar check on the direct cutoff on x, as that way you do not examine the case whether the assignment is certain or not.

It is up to you to decide whether your application needs the information about a gray zone or how close the decision was, and whether this is better achieved via the logistic regressions predicted probabiliy or directly by x (e.g. a medical doctor* may find the x information more useful).

* assuming that is who gets medical biomarker results. Btw, over here (Germany), only approbated medical doctor are allowed to actually give a diagnosis. Thus, clinical lab results are e.g. reported as x together with its cutoff (normal range) and the summary information whether it is inside/outside the normal range/positive or negative (depending on the marker/analyte).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.