I have a very limited data set with a number of features $\textbf{x}$ (roughly 30 dimensions) and a regression target $y$. I have roughly $20$ groups of data-points with $2$ points in each group (where each group corresponds to a different participant in an experiment). I wish to perform a linear regression to determine the most useful features to predict the target and I want some measure of how confident I am that these features are useful. I also want some idea of how reliable a prediction I can make given my model.

Looking elsewhere, I see that step-wise regression is not advised, and that instead LASSO is reasonably well regarded. I am also aware of other methods: ElasticNet and horseshoe and double pareto priors, but I want to keep things straightforward for a first analysis. See here, here and here.

Because the data is limited, I realise that I won't be able to make any strong claims, but I want to have something that is indicative and motivates collection of a larger data-set (perhaps focused on more relevant features). What I therefore propose is the following:

  1. Use cross-validation to score a selection of regularisation values $\lambda$ for the LASSO, to determine the best. For this, I will shuffle the whole data randomly. Perhaps 10-fold cross validation is sufficient here.
  2. Given a good $\lambda$, fit the whole of the data to determine the r-squared value, and to inspect which features are selected (not give $0$ weight).
  3. To get a better idea of performance predicting unseen participants, I will re-partition the data such that each partition contains two data-points from a single participant. Perform cross-validation to determine predicted r-squared using the $\lambda$ value from before.

Some issues I have found with this strategy are:

  • When cross-validating with very small test sets, calculating predicted r-squared on each fold and then averaging leads to very unstable predictions. Instead, I intend to collect the target-prediction pairs over all folds, and calculate my r-squared once on the collected data. Does that make sense?
  • In step 2., if I want to compare the resulting model with one from another procedure, then I would like to use something like AIC or adjusted r-squared, but how many parameters should I say my model has? The original 30, or the number of non-zero weights in the model?
  • In step 3., the LASSO could be selecting different features in each fold, so how do I determine which features are interesting? Should I look at empirical confidence intervals for the weights across all folds?

Does this sound at all reasonable? Or am I being hopelessly naive, or missing an obvious alternative?

  • $\begingroup$ Regarding cross validation, according to some (e.g. Frank Harrell) a better alternative is in general the Efron-Gong optimism bootstrap. Look it up, it's easy to implement, faster and less noisy. $\endgroup$ – Gino_JrDataScientist Jun 7 '18 at 10:38

You've thought this through very well and raised important questions. Besides bootstrapping the entire process as Gino stated, take a step back and look at the complexity of the task vs. the information content available to the task. Your effective sample size is just over 20, so the probability that any feature selection method will select the 'right' features is zero, when the number of candidate features is 30. Consider instead visualizing the problem as one of "which of the 30 features are hard to separate from each other, and if we didn't separate them, how do groups of features relate to outcome?". In other words, consider the use of data reduction using unsupervised learning techniques such as variable clustering and principle components. Sparse principal components, as demonstrated in my RMS book is also worth a look.

The overall strategy with data reduction is to reduce the $X$ space down to a number of dimensions that your data will likely support in predicting $Y$. For your case I would be surprised if you can afford to have more than two parameters to estimate to predict $Y$, besides an intercept.

You also need to take into account within-subject correlation.

  • $\begingroup$ Would you recommend the use of dimensionality reduction on the features independent of the target, e.g. PCA or similar, or does it make more sense to look at regression techniques that "group" features within the context of the regression task, e.g. ElasticNet or Ordered Weighted l1 Regularisation (OWL) (e.g. arxiv.org/abs/1409.4005)? Also, when you refer to within-subject correlation, could you expand a little on how this might affect things, or need to be adjusted for? $\endgroup$ – pandamonium Jun 7 '18 at 11:43
  • $\begingroup$ Yes for your situation I'd go with unsupervised data reduction so the predictive task is easier. Once you reduce dimensions down into some sort of feature scores (PCs or otherwise) or remove redundant features (features that are easily predicted from all the other features - see the R Hmisc redun function), you can use a model made for repeated measures such as a mixed efforts Bayesian hierarchical model or a fixed effects frequentist model. Or adjust standard errors after-the-fit using the robust cluster sandwich covariance estimator (e.g., R rms package robcov function). $\endgroup$ – Frank Harrell Jun 7 '18 at 11:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.