# When is it valid to use lasso and adaptive lasso

I am brand new to LASSO. I have a problem in that I have a data set without about 440 usable cases only about 42 of them of one level of the DV (every level of the DV and predictor have two levels). I have 39 predictors and my understanding of logistic regression is that is simply too many for only this many cases. But I have had trouble using LASSO (or I am not sure I can use it anyhow given what I encountered). This is in SAS, I don't know R well enough to do this in that code. I split the total data set into two pieces one to choose the variables with lasso and a second to run the selected variables. There are about 230 usable cases for each data set about 10 percent at one level of the DV in each. The SAS code (I can not send the data because I work for a state agency) is:

ODS graphics on;
proc glmselect data=randomdata plots=all;
partition fraction(validate=.3);
class pd1 pd2 pd3 pd4 pd5 pd6 pd7 pd8 pd9 pd10 pd11  pd12 pd13 pd14 pd15 pd16 pd17
pd18 pd19 pd20 pd21 pd22 pd23 pd24 pd25 pd26 pd27 pd28 pd29 pd30 pd31 ;
model dvd = pd1 pd2 pd3 pd4 pd5 pd6 pd7 pd8 pd9 pd10 pd11 pd12 pd13 pd14 pd15 pd16 pd17
pd18 pd19 pd20 pd21 pd22 pd23 pd24 pd25 pd26 pd27 pd28 pd29 pd30 pd31
/ selection=lasso(stop=none choose=validate);
run;

proc glmselect data=randomdata plots=all;
partition fraction(validate=.3);
class pd1 pd2 pd3 pd4 pd5 pd6 pd7 pd8 pd9 pd10 pd11 pd12 pd13 pd14 pd15 pd16 pd17
pd18 pd19 pd20 pd21 pd22 pd23 pd24 pd25 pd26 pd27 pd28 pd29 pd30 pd31 ;
model dvd = pd1 pd2 pd3 pd4 pd5 pd6 pd7 pd8 pd9 pd10 pd11 pd12 pd13 pd14 pd15 pd16 pd17
pd18 pd19 pd20 pd21 pd22 pd23 pd24 pd25 pd26 pd27 pd28 pd29 pd30 pd31
run;

ods graphics off;


When I run this code I notice two warnings.

WARNING: The adaptive weights for the LASSO method are not uniquely determined because the full least squares model is singular.

and then in the Output:

Selection stopped because all candidate effects for entry are linearly dependent on effects in the model.

And the logistic regression I tested against the reduced data base gets the following warning:

There is a complete separation of data points. The maximum likelihood estimate does not exist.

I am not using this half of the data to estimate any regression just to choose variables with lasso/adaptive lasso. I use the other half of the original data held out and the list of variables to run logistic regression and ran into no issues of any kind. But I don't know if its valid to use the variables selected by lasso/adaptive lasso when you run into this issue - that is will lasso and adaptive lasso work when these issues occur.

• You have far too few cases to do a train/test split like you are. Better to fit a model to all the data and then evaluate the consistency and reproducibility of the variable selection by bootstrapping. Also, what are you trying to accomplish with the "adaptive lasso"? See the freely available Statistical Learning with Sparsity, section 4.6 on adaptive lasso and Chapter 6, for a bootstrap example. Or include all reasonable predictors with ridge regression.
– EdM
Sep 20 at 19:42
• I am trying to chose a reasonable set of predictors to be run in logistic regression. I thought a primary point of LASSO was that it was usable with sparse data. My understanding is that adaptive LASSO was better at choosing the most correct set of predictors than LASSO which is why I am using it. My reading of the literature suggested that. Similarly for choosing the right predictors my understanding is that LASSO is much better. Ridge does not remove any variable form the model, it is used I thought to deal with MC. Sep 20 at 22:37
• As I reviewed what you said EdM I have a question. Are you saying to use the whole data base to choose the variables with LASSO (so don't hold out half the data base to later run the logistic regression on) and then use the variables the LASSO selects on that data base? I have to split a data base by train/test to choose the alpha level but I can do it on the entire data base not the half I split off. Thanks for the book. I have been looking for every source I can find on line, but they are limited in terms of detail. Sep 20 at 23:22
• Even using the whole data set to build LASSO I ran into the same errors which is strange since when I ran logistic regression on this originally (before I tried to use LASSO to reduce the number of variables) I got no indication of a problem. Sep 20 at 23:47

A frequent confusion in this type of modeling is that much of what you read about "machine learning" works very well when there are tens of thousands or cases and hundreds to thousands of predictors or more, but can get you into trouble with data sets of this size.

First, this is far too small a data set to set aside separate subsets for predictor selection and model development. At this scale, build the model with the whole data set and validate as best as you can internally by repeated modeling on bootstrap samples from the data. Section 6.2 of Statistical Learning with Sparsity (SLS) shows how to do this with LASSO modeling.

Second, you have to have a clear view of why you want to do predictor selection.

(a) If your interest is in predicting new cases and all of your 39 predictors will be available for new cases, ridge regression allows you to use all the available information in a principled way (with penalization that should also remove your complete separation problem). This is different, say, from gene-expression studies where you have 20,000 potential predictors and have to reduce the predictor numbers to a substantially smaller set of a few dozen to allow reasonably affordable clinical tests. It's also possible to use your knowledge of the subject matter to combine related predictors in a way that doesn't depend on knowing their relationships to outcome. If some predictors are unlikely to be available in the future and you are trying to do prediction, why include them in the model at all? See Chapter 4 of Frank Harrell's course notes or book for valuable guidance on many such aspects of regression and predictor selection, guidance that can be applied before you jump into LASSO.

(b) If you are doing predictor selection to identify the "most important" predictors, you are likely to be disappointed in the results. With about the same number of predictors as you have members of the minority outcome class, your predictors are likely to be highly inter-correlated. LASSO might select one such predictor from a correlated set on these data, a completely different predictor from that correlated set on a new data sample. You can see that in modeling on bootstrap samples, as shown in Section 6.2 of SLS, cited above.

Third, you have to be very careful that the approaches you are using are suitable for your application. The warning from adaptive LASSO that "adaptive weights for the LASSO method are not uniquely determined because the full least squares model is singular" suggests that the software was trying to do a least-squares fit on your binary outcomes. That's not a good approach. I have no experience with adaptive LASSO and don't know how well it works (if at all) with binary outcomes. If you are wedded to adaptive LASSO and it can be applied to binary outcomes, you should set the initial weights based on single-predictor logistic regressions instead, generalizing the recommendation on page 86 of SLS for the linear regression case.

My understanding is that adaptive LASSO is designed to give even more sparsity in retained predictor numbers than regular LASSO, which doesn't seem wise in this case. With 42 members of the minority outcome class, a standard logistic regression LASSO with penalty optimized via cross-validated deviance should return about 3 or 4 predictors--if predictor selection is really what you need.

• This came about because I felt we had too many predictors for a logistic regression model (although the model ran). We had 39 predictors and only about 45 cases at one level of the DV (out of about 445 total). From what I have read, e.g., agressti, it is unwise to have more than one predictor for every ten cases. We have no theory at all to throw out some predictors. A researcher I know said for sparse data like this LASSO would be a good way to reduce the list of variables. One article I read suggested that the fact the DV was categorial would not matter for this. But the lasso won't run. Sep 26 at 1:50
• I wanted to point out that I ran a VIF, using linear regression, but I don' this matters for testing multicollinearity, for these variables and no VIF was higher than 3 while the normal warning level is 5, or some say 10. It is true that many of the 39 variables are not statistically significant. My concern was not multicollinearity, which is why I understand you use Ridge, but simply have too many predictors for the data. I thought lasso was preferred to ridge in selecting variables. I am not weeded to adaptive lasso, but the fact that it shrinks the smaller slopes more seemed to make sense. Sep 26 at 1:59
• The sas product I am using always builds linear models in generating lasso. Again what I read that commented on this said this was not a big issue if using it just to select variables. But from your comments I don't think you agree. So in the future I am not going to do this. I do not know R well enough to utilize the R modules for lasso. I use it for limited things, mainly time series. I am trying to learn it more - historically I used only SAS. By the way thank you for your comments. Sep 26 at 2:08
• @user54285 SAS/STAT has implemented LASSO for logistic regression since version 14.1 in the HPGENSELECT procedure. Ridge can help with problems besides multicollinearity: it penalizes coefficients to reduce overfitting and to avoid "complete separation" problems. The R LASSO is nicely illustrated in Section 6.6.2 of ISLR, ridge in Section 6.6.1; although shown for a Gaussian model, you just specify family="binomial" for logistic models.
– EdM
Sep 26 at 14:28