2
$\begingroup$

I have a database with 200+ variables and less then 50 cases. I need to choose an optimal model that predicts one dependent variable.

Are stepwise/lasso regressions still appropriate methods to build such model?

$\endgroup$
  • $\begingroup$ I would say "no" since you don't have enough cases to cross validate and stepwise takes enormous advantage of chance with so many variables. $\endgroup$ – David Lane May 6 '17 at 23:08
2
$\begingroup$

It looks like your objective is prediction, in which case stepwise might not be too bad but be wary of calling the resulting model 'optimal' unless you're clear about what 'optimal' actually means to you in this case. Even still, I'd say the LASSO is a better option.

I would strongly advocate a Bayesian approach to these 'variable selection' problems if you want to use regression. You really need a very full picture of the uncertainty in your estimates in these situations and the usual confidence intervals etc. are not really appropriate. There are Bayesian versions of the LASSO (including fused/grouped LASSO) as well as the Horseshoe prior, which is supposedly better at separating strong and weak signals (large effects remain unshrunk, which is not the case with the LASSO). You can implement these methods fairly straightforwardly using the R packages rstanarm and/or brms if you're doing (generalised) linear modelling. You can also use the package loo to perform approximate leave one out cross validation for model comparison, which might be appropriate here.

You could try a non-parametric approach like a regression forest or a boosting machine, both of which can be implemented in caret (along with literally hundreds of other methods) and provide 'variable importance' measures.

You should not do any of this blindly, though. Don't just throw this at glmnet and take the results as gospel, think about whether your results make sense. If they don't, there are possibly some variables you can discount before even starting -- if you get results that make you think, "well, that predictor should definitely not be important", just scrap that predictor to begin with.

Tibshirani and Hastie are the main frequentist authors for this stuff (see their papers and especially their book 'Elements of Statistical Learning'). For Bayesian, I'd recommend starting from the Horseshoe paper.

$\endgroup$
1
$\begingroup$

Yes. This is called a "$p > n$" problem sometimes. (Here, $p$ is the number of covariates and $n$ is the sample size.)

You can do stepforward regression, but not stepback (since the least squares fit on all of the covariates is not unique.) This setup is perfect for lasso. There's even theoretical support (under some conditions) for the lasso in this setting.

Unfortunately, all of these methods are (somewhat) limited since they'll select at most $n$ covariates to have nonzero estimated coefficient. (see, for instance, here.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.