For LASSO, what is the best alternative when a held out test set is not viable?

Question

I have a very limited data set with a number of features $\textbf{x}$ (roughly 30 dimensions) and a regression target $y$. I have roughly $20$ groups of data-points with $2$ points in each group (where each group corresponds to a different participant in an experiment). I wish to perform a linear regression to determine the most useful features to predict the target and I want some measure of how confident I am that these features are useful. I also want some idea of how reliable a prediction I can make given my model.

Looking elsewhere, I see that step-wise regression is not advised, and that instead LASSO is reasonably well regarded. I am also aware of other methods: ElasticNet and horseshoe and double pareto priors, but I want to keep things straightforward for a first analysis. See here, here and here.

Because the data is limited, I realise that I won't be able to make any strong claims, but I want to have something that is indicative and motivates collection of a larger data-set (perhaps focused on more relevant features). What I therefore propose is the following:

1. Use cross-validation to score a selection of regularisation values $\lambda$ for the LASSO, to determine the best. For this, I will shuffle the whole data randomly. Perhaps 10-fold cross validation is sufficient here.
2. Given a good $\lambda$, fit the whole of the data to determine the r-squared value, and to inspect which features are selected (not give $0$ weight).
3. To get a better idea of performance predicting unseen participants, I will re-partition the data such that each partition contains two data-points from a single participant. Perform cross-validation to determine predicted r-squared using the $\lambda$ value from before.

Some issues I have found with this strategy are:

• When cross-validating with very small test sets, calculating predicted r-squared on each fold and then averaging leads to very unstable predictions. Instead, I intend to collect the target-prediction pairs over all folds, and calculate my r-squared once on the collected data. Does that make sense?
• In step 2., if I want to compare the resulting model with one from another procedure, then I would like to use something like AIC or adjusted r-squared, but how many parameters should I say my model has? The original 30, or the number of non-zero weights in the model?
• In step 3., the LASSO could be selecting different features in each fold, so how do I determine which features are interesting? Should I look at empirical confidence intervals for the weights across all folds?

Does this sound at all reasonable? Or am I being hopelessly naive, or missing an obvious alternative?

Answer 1

You've thought this through very well and raised important questions. Besides bootstrapping the entire process as Gino stated, take a step back and look at the complexity of the task vs. the information content available to the task. Your effective sample size is just over 20, so the probability that any feature selection method will select the 'right' features is zero, when the number of candidate features is 30. Consider instead visualizing the problem as one of "which of the 30 features are hard to separate from each other, and if we didn't separate them, how do groups of features relate to outcome?". In other words, consider the use of data reduction using unsupervised learning techniques such as variable clustering and principle components. Sparse principal components, as demonstrated in my RMS book is also worth a look.

The overall strategy with data reduction is to reduce the $X$ space down to a number of dimensions that your data will likely support in predicting $Y$. For your case I would be surprised if you can afford to have more than two parameters to estimate to predict $Y$, besides an intercept.

You also need to take into account within-subject correlation.