# For LASSO, what is the best alternative when a held out test set is not viable?

I have a very limited data set with a number of features $\textbf{x}$ (roughly 30 dimensions) and a regression target $y$. I have roughly $20$ groups of data-points with $2$ points in each group (where each group corresponds to a different participant in an experiment). I wish to perform a linear regression to determine the most useful features to predict the target and I want some measure of how confident I am that these features are useful. I also want some idea of how reliable a prediction I can make given my model.

Looking elsewhere, I see that step-wise regression is not advised, and that instead LASSO is reasonably well regarded. I am also aware of other methods: ElasticNet and horseshoe and double pareto priors, but I want to keep things straightforward for a first analysis. See here, here and here.

Because the data is limited, I realise that I won't be able to make any strong claims, but I want to have something that is indicative and motivates collection of a larger data-set (perhaps focused on more relevant features). What I therefore propose is the following:

1. Use cross-validation to score a selection of regularisation values $\lambda$ for the LASSO, to determine the best. For this, I will shuffle the whole data randomly. Perhaps 10-fold cross validation is sufficient here.
2. Given a good $\lambda$, fit the whole of the data to determine the r-squared value, and to inspect which features are selected (not give $0$ weight).
3. To get a better idea of performance predicting unseen participants, I will re-partition the data such that each partition contains two data-points from a single participant. Perform cross-validation to determine predicted r-squared using the $\lambda$ value from before.

Some issues I have found with this strategy are:

• When cross-validating with very small test sets, calculating predicted r-squared on each fold and then averaging leads to very unstable predictions. Instead, I intend to collect the target-prediction pairs over all folds, and calculate my r-squared once on the collected data. Does that make sense?
• In step 2., if I want to compare the resulting model with one from another procedure, then I would like to use something like AIC or adjusted r-squared, but how many parameters should I say my model has? The original 30, or the number of non-zero weights in the model?
• In step 3., the LASSO could be selecting different features in each fold, so how do I determine which features are interesting? Should I look at empirical confidence intervals for the weights across all folds?

Does this sound at all reasonable? Or am I being hopelessly naive, or missing an obvious alternative?

• Regarding cross validation, according to some (e.g. Frank Harrell) a better alternative is in general the Efron-Gong optimism bootstrap. Look it up, it's easy to implement, faster and less noisy. – Gino_JrDataScientist Jun 7 '18 at 10:38

The overall strategy with data reduction is to reduce the $X$ space down to a number of dimensions that your data will likely support in predicting $Y$. For your case I would be surprised if you can afford to have more than two parameters to estimate to predict $Y$, besides an intercept.
• Yes for your situation I'd go with unsupervised data reduction so the predictive task is easier. Once you reduce dimensions down into some sort of feature scores (PCs or otherwise) or remove redundant features (features that are easily predicted from all the other features - see the R Hmisc redun function), you can use a model made for repeated measures such as a mixed efforts Bayesian hierarchical model or a fixed effects frequentist model. Or adjust standard errors after-the-fit using the robust cluster sandwich covariance estimator (e.g., R rms package robcov function). – Frank Harrell Jun 7 '18 at 11:53