I'm comparing two models of classification of a binary medical condition. number of patients = 125 (95 and 30 for each condition)

  • model 1 = radiologist : obtained with logistic regression with "visual predictors" (5, in which 2 multimodal categorical variables)
  • model 2 = radiologist+ : obtained with logistic regression using the 5 previous "visual predictors" and 7 quantitative predictors (image textural and histogram variables). The 7 histo/textural predictors were obtained from Lasso logistic regression (best lambda after 5 fold cross validation)

Do you think it's reasonable to have this approach ? I read in p158 of Statistical Learning with Sparsity that is possible to make post-selection inference on stepwise forward regression, but it seems to be kludge to have my approach.

I used this approach also because I'd never did a grouped Lasso, and as I have manny categorical multimodal variable, it was easier to manage them with log reg. If you have the code R for groupes Lasso logistic regression than I would be very grateful if any one share it with me.

Also, the aim behind using Lasso only for histo/textural quantitative parameters is because I wanted to keep the same baseline "usual radiologist predictors" as I want to analyse the effect of this addition to the "usual" diagnostic performances.

Thanks for your help


If you use LASSO to choose predictors and then use those predictors in an unpenalized logistic regression, your p-values etc are incorrect because you haven't taken into account your use of the data to select the predictors. See this answer for the identical situation in least-squares regression. As discussed there, accounting for that pre-selection of predictors is what you call the "kludge" is all about for inference on variable-selection approaches like stepwise and LASSO.

The main problem you face with your data is too few cases. With only 30 in the minority class, an unpenalized model (regular logistic regression) would need to be limited to only about 2 predictors to avoid overfitting. The usual rule of thumb is 15 minority-class cases per predictor, regardless of which class is considered the target class. A 3-level categorical predictor, for example, would already count as 2 predictors in terms of overfitting.

You need some type of penalized approach to deal with this. LASSO, keeping the penalized values of the logistic regression coefficients, could work in principle but would probably only return a very few predictors. If the main interest is in prediction you should seriously consider ridge regression, combining information from all data while avoiding overfitting by penalizing the predictors that don't add much information.

One approach to consider for comparing the "usual radiological predictors" alone versus adding features by some defined procedure would be to compare the results side-by-side on multiple bootstrapped re-samples of your data, modeling on the re-samples and testing on the complete data set. Then you use the standard errors of the performance estimates returned by the multiple models with both approaches to look for "significant" differences between the 2 approaches without needing the "kluge." This answer outlines how you might proceed.

I suppose that there is no reason why you couldn't try to use your LASSO pre-selection and subsequent logistic modeling as the "defined procedure" for adding predictors, as in the prior paragraph. You would have to repeat all steps of the procedure--the LASSO for variable selection and subsequent logistic regression--within each of the bootstrapped samples, testing against the full data set. My guess is that you wouldn't get very satisfactory performance, but if you do you could then use the bootstrap results to document the quality of your modeling approach.

  • $\begingroup$ Thank you for your response. Actually the targeted condition is the one with 95 cases. but as you said, if we consider that we should have 10-15 observations per predictor including the dummy one. knowing this limits, I would have your point of view for the main questioning I have: is it reasonable (ignoring the fact that in my case there will be clearly overfitting) to use Lasso as feature selection procedure and re-inject the quantitative to a "baseline" logistic regression model ? $\endgroup$ – Miricy Sep 7 '20 at 20:14
  • 1
    $\begingroup$ @Miricy what matters for overfitting is the number of cases in the smaller group, regardless of which is the "targeted condition." Will add a bit to the answer to respond to the rest of your comment. $\endgroup$ – EdM Sep 7 '20 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.