How to use LASSO to select glm model gaussian

Question

I have a small sample size $n<20$. I want to find which combination of 8 variables better predict $y$.

I was using a stepAICc but it is suggested to away stepwise model selection. I have tried lars and glmnet but I don't understand the output. Before with stepAICc I could just pick the model with the lowest AICc value, but how do I proceed with lasso? Output from glmnet is

Call:  glmnet(x = x, y = y, family = "gaussian") 

       Df    %Dev    Lambda
  [1,]  0 0.00000 3.416e-01
  [2,]  1 0.09574 3.113e-01
  [3,]  1 0.17520 2.836e-01
  [4,]  1 0.24120 2.584e-01
  [5,]  2 0.29650 2.355e-01
  [6,]  2 0.34420 2.146e-01
  [7,]  2 0.38380 1.955e-01
  [8,]  2 0.41660 1.781e-01
  [9,]  2 0.44390 1.623e-01
 [10,]  2 0.46650 1.479e-01

And output from lars

Call:
lars(x = x, y = y)
R-squared: 0.76 
Sequence of LASSO moves:
     st0011sme ss0011sme bs0011yme ss0011yme st0011yme st0011sme bt0011sme st0011sme bt0011yme bs0011sme bt0011yme
Var          3         7         6         8         4        -3         1         3         2         5        -2
Step         1         2         3         4         5         6         7         8         9        10        11
     ss0011yme ss0011yme bt0011yme bt0011sme bt0011sme
Var         -8         8         2        -1         1
Step        12        13        14        15        16
> summary(las)
LARS/LASSO
Call: lars(x = x, y = y)
   Df    Rss      Cp
0   1 3.3117 15.1324
1   2 2.3602  8.7622
2   3 1.4104  2.4066
3   4 1.3931  4.2552
4   5 1.1681  4.2758
5   6 1.1502  6.1177

Answer 1

The task is impossible, which can be revealed by bootstrapping the entire modeling and feature selection process to show that

1. Confidence intervals on importance rankings of the candidate predictors will be roughly from 1-8 for all 8 candidates
2. The features selected will vary wildly over bootstrap replications

Essentially the data do not have the information needed to choose which features are the "right features" no matter which feature selection method you are using. Reliable feature selection requires huge sample sizes, absence of collinearities, and large numbers of predictors actually having true coefficients of zero.

For your problem don't try to "name names". Just fit a penalized model with all candidate predictors, or get a much larger sample size. It requires $n=70$ just to estimate the residual standard deviation in a linear model. More details are in my RMS book and course notes.

Answer 2

As mentioned, with the LASSO approach it is common to use cross-validation to select the fixed lambda value. Most people default to selecting the value of lambda within 1 SE of the used criteria (e.g., AUC). This is referred to as the most regularized model. I believe it was initially recommended by Friedman.

Your follow-up question, what about your small sample. Well you definitely have a small sample and potential threats of sparsity. A general rule (sorry I don't have a citation) is that the smaller the sample the more folds you should use in cross-validation. This is to ensure greater independence between folds. However, you have a fairly small set so switching from k= 5 to 10 would mean silly small sets. I haven't seen simulation studies examining this. We know LASSO does well in p > n scenarios. Though we would want to question whether a model based on a sample of say 2 observations could accurately represent the overall population of interest and Gaussian assumptions?

At this point I would ask whether or not your a prior domain knowledge could not come into play in regards to the feature selection. Or perhaps the use of Bayesian modeling/regularization given the small sample. I would note that your final results are going to be conditional on the modeling processs and the more models you run and switching approaches, the greater the risk for selective type I errors.

P.S., Per your above output (top pane), presented in the degrees of freedom allude to number of terms in model, % deviance, and the associated lambda value for that model. You can also plot these data for a better interpretation of what is going on.