When is it valid to use lasso and adaptive lasso

Question

I am brand new to LASSO. I have a problem in that I have a data set without about 440 usable cases only about 42 of them of one level of the DV (every level of the DV and predictor have two levels). I have 39 predictors and my understanding of logistic regression is that is simply too many for only this many cases. But I have had trouble using LASSO (or I am not sure I can use it anyhow given what I encountered). This is in SAS, I don't know R well enough to do this in that code. I split the total data set into two pieces one to choose the variables with lasso and a second to run the selected variables. There are about 230 usable cases for each data set about 10 percent at one level of the DV in each. The SAS code (I can not send the data because I work for a state agency) is:

ODS graphics on;
proc glmselect data=randomdata plots=all;
partition fraction(validate=.3);
class pd1 pd2 pd3 pd4 pd5 pd6 pd7 pd8 pd9 pd10 pd11  pd12 pd13 pd14 pd15 pd16 pd17
pd18 pd19 pd20 pd21 pd22 pd23 pd24 pd25 pd26 pd27 pd28 pd29 pd30 pd31 ;
model dvd = pd1 pd2 pd3 pd4 pd5 pd6 pd7 pd8 pd9 pd10 pd11 pd12 pd13 pd14 pd15 pd16 pd17
pd18 pd19 pd20 pd21 pd22 pd23 pd24 pd25 pd26 pd27 pd28 pd29 pd30 pd31
/ selection=lasso(stop=none choose=validate);
run;

proc glmselect data=randomdata plots=all;
partition fraction(validate=.3);
class pd1 pd2 pd3 pd4 pd5 pd6 pd7 pd8 pd9 pd10 pd11 pd12 pd13 pd14 pd15 pd16 pd17
pd18 pd19 pd20 pd21 pd22 pd23 pd24 pd25 pd26 pd27 pd28 pd29 pd30 pd31 ;
model dvd = pd1 pd2 pd3 pd4 pd5 pd6 pd7 pd8 pd9 pd10 pd11 pd12 pd13 pd14 pd15 pd16 pd17
pd18 pd19 pd20 pd21 pd22 pd23 pd24 pd25 pd26 pd27 pd28 pd29 pd30 pd31
/ selection=lasso(adaptive stop=none choose=validate);
run;

ods graphics off;

When I run this code I notice two warnings.

WARNING: The adaptive weights for the LASSO method are not uniquely determined because the full least squares model is singular.

and then in the Output:

Selection stopped because all candidate effects for entry are linearly dependent on effects in the model.

And the logistic regression I tested against the reduced data base gets the following warning:

There is a complete separation of data points. The maximum likelihood estimate does not exist.

I am not using this half of the data to estimate any regression just to choose variables with lasso/adaptive lasso. I use the other half of the original data held out and the list of variables to run logistic regression and ran into no issues of any kind. But I don't know if its valid to use the variables selected by lasso/adaptive lasso when you run into this issue - that is will lasso and adaptive lasso work when these issues occur.

Answer 1

A frequent confusion in this type of modeling is that much of what you read about "machine learning" works very well when there are tens of thousands or cases and hundreds to thousands of predictors or more, but can get you into trouble with data sets of this size.

First, this is far too small a data set to set aside separate subsets for predictor selection and model development. At this scale, build the model with the whole data set and validate as best as you can internally by repeated modeling on bootstrap samples from the data. Section 6.2 of Statistical Learning with Sparsity (SLS) shows how to do this with LASSO modeling.

Second, you have to have a clear view of why you want to do predictor selection.

(a) If your interest is in predicting new cases and all of your 39 predictors will be available for new cases, ridge regression allows you to use all the available information in a principled way (with penalization that should also remove your complete separation problem). This is different, say, from gene-expression studies where you have 20,000 potential predictors and have to reduce the predictor numbers to a substantially smaller set of a few dozen to allow reasonably affordable clinical tests. It's also possible to use your knowledge of the subject matter to combine related predictors in a way that doesn't depend on knowing their relationships to outcome. If some predictors are unlikely to be available in the future and you are trying to do prediction, why include them in the model at all? See Chapter 4 of Frank Harrell's course notes or book for valuable guidance on many such aspects of regression and predictor selection, guidance that can be applied before you jump into LASSO.

(b) If you are doing predictor selection to identify the "most important" predictors, you are likely to be disappointed in the results. With about the same number of predictors as you have members of the minority outcome class, your predictors are likely to be highly inter-correlated. LASSO might select one such predictor from a correlated set on these data, a completely different predictor from that correlated set on a new data sample. You can see that in modeling on bootstrap samples, as shown in Section 6.2 of SLS, cited above.

Third, you have to be very careful that the approaches you are using are suitable for your application. The warning from adaptive LASSO that "adaptive weights for the LASSO method are not uniquely determined because the full least squares model is singular" suggests that the software was trying to do a least-squares fit on your binary outcomes. That's not a good approach. I have no experience with adaptive LASSO and don't know how well it works (if at all) with binary outcomes. If you are wedded to adaptive LASSO and it can be applied to binary outcomes, you should set the initial weights based on single-predictor logistic regressions instead, generalizing the recommendation on page 86 of SLS for the linear regression case.

My understanding is that adaptive LASSO is designed to give even more sparsity in retained predictor numbers than regular LASSO, which doesn't seem wise in this case. With 42 members of the minority outcome class, a standard logistic regression LASSO with penalty optimized via cross-validated deviance should return about 3 or 4 predictors--if predictor selection is really what you need.