Is it reasonable to use Lasso as selection features approach before logistic regression

Question

I'm comparing two models of classification of a binary medical condition. number of patients = 125 (95 and 30 for each condition)

• model 1 = radiologist : obtained with logistic regression with "visual predictors" (5, in which 2 multimodal categorical variables)
• model 2 = radiologist+ : obtained with logistic regression using the 5 previous "visual predictors" and 7 quantitative predictors (image textural and histogram variables). The 7 histo/textural predictors were obtained from Lasso logistic regression (best lambda after 5 fold cross validation)

Do you think it's reasonable to have this approach ? I read in p158 of Statistical Learning with Sparsity that is possible to make post-selection inference on stepwise forward regression, but it seems to be kludge to have my approach.

I used this approach also because I'd never did a grouped Lasso, and as I have manny categorical multimodal variable, it was easier to manage them with log reg. If you have the code R for groupes Lasso logistic regression than I would be very grateful if any one share it with me.

Also, the aim behind using Lasso only for histo/textural quantitative parameters is because I wanted to keep the same baseline "usual radiologist predictors" as I want to analyse the effect of this addition to the "usual" diagnostic performances.

Thanks for your help

Answer 1

If you use LASSO to choose predictors and then use those predictors in an unpenalized logistic regression, your p-values etc are incorrect because you haven't taken into account your use of the data to select the predictors. See this answer for the identical situation in least-squares regression. As discussed there, accounting for that pre-selection of predictors is what you call the "kludge" is all about for inference on variable-selection approaches like stepwise and LASSO.

The main problem you face with your data is too few cases. With only 30 in the minority class, an unpenalized model (regular logistic regression) would need to be limited to only about 2 predictors to avoid overfitting. The usual rule of thumb is 15 minority-class cases per predictor, regardless of which class is considered the target class. A 3-level categorical predictor, for example, would already count as 2 predictors in terms of overfitting.

You need some type of penalized approach to deal with this. LASSO, keeping the penalized values of the logistic regression coefficients, could work in principle but would probably only return a very few predictors. If the main interest is in prediction you should seriously consider ridge regression, combining information from all data while avoiding overfitting by penalizing the predictors that don't add much information.

One approach to consider for comparing the "usual radiological predictors" alone versus adding features by some defined procedure would be to compare the results side-by-side on multiple bootstrapped re-samples of your data, modeling on the re-samples and testing on the complete data set. Then you use the standard errors of the performance estimates returned by the multiple models with both approaches to look for "significant" differences between the 2 approaches without needing the "kluge." This answer outlines how you might proceed.

I suppose that there is no reason why you couldn't try to use your LASSO pre-selection and subsequent logistic modeling as the "defined procedure" for adding predictors, as in the prior paragraph. You would have to repeat all steps of the procedure--the LASSO for variable selection and subsequent logistic regression--within each of the bootstrapped samples, testing against the full data set. My guess is that you wouldn't get very satisfactory performance, but if you do you could then use the bootstrap results to document the quality of your modeling approach.