Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In a dataset of two non-overlapping populations (patients & healthy, total $n=60$) I would like to find (out of $300$ independent variables) significant predictors for a continuous dependent variable. Correlation between predictors is present. I am interested in finding out if any of the predictors are related to the dependent variable "in reality" (rather than predicting the dependent variable as exactly as possible). As I got overwhelmed with the numerous possible approaches, I would like to ask for which approach is most recommended.

  • From my understanding stepwise inclusion or exclusion of predictors is not recommended

  • E.g. run a linear regression separately for every predictor and correct p-values for multiple comparison using FDR (probably very conservative?)

  • Principal-component regression: difficult to interpret as I won't be able to tell about the predictive power of individual predictors but only about the components.

  • any other suggestions?

share|improve this question
I've heard of people using L1 regularized regression to do this type of things. But I do not know enough to write a proper answer... – King Aug 21 '12 at 12:45
In order to give the best recommendations, it would help us to know how you will proceed after identifying "significant predictors." Are you trying to predict the outcome as exactly as possible; find a parsimonious way to predict it (e.g., using a set of up to k predictors that will efficiently do so; explain what causes the outcome "in reality"; or something else? Also, how big is your data set? – rolando2 Aug 21 '12 at 15:40
@rolando: thanks for the comment! I updated the question: my total number of observations is n=60 subjects. My aim is not to predict the dependent variable as exactly as possible but rather to explain what causes the outcome "in reality" (= hope to find relationsship between variables that could be confirmed in later studies/datasets) – jokel Aug 21 '12 at 21:16
I also posted a follow-up question including some dummy data. I would be very thankful for all hints.… – jokel Aug 22 '12 at 15:40
up vote 28 down vote accepted

I would recommend trying a glm with lasso regularization. This adds a penalty to the model for number of variables, and as you increase the penalty, the number of variables in the model will decrease.

You should use cross-validation to select the value of the penalty parameter. If you have R, I suggest using the glmnet package. Use alpha=1 for lasso regression, and alpha=0 for ridge regression. Setting a value between 0 and 1 will use a combination of lasso and ridge penalties, also know as the elastic net.

share|improve this answer
I agree with Zach. David Cassell and I wrote a paper about this, concentrating on SAS but not entirely. It is Stopping Stepwise. – Peter Flom Aug 21 '12 at 13:24
I think it's 0 for ridge and 1 for lasso – King Aug 21 '12 at 14:01
@Zach: Thanks for the hints. Is there a way to obtain some test-statistic that would allow me to judge the significance of single predictors. In the end I would like to be able to say "predictor X is significantly related to dependent variable Y". – jokel Aug 22 '12 at 10:22
Regarding CIs, from the manual of another R package implementing the LASSO (…, page 18): "It is a very natural question to ask for standard errors of regression coefficients or other estimated quantities. In principle such standard errors can easily be calculated, e.g. using the bootstrap. Still, this package deliberately does not provide them. The reason for this is that standard errors are not very meaningful for strongly biased estimates such as arise from penalized estimation methods." – miura Sep 19 '12 at 9:01
@miura Recently introduced was a test statistics for just that, by the original Lasso authors too: paper and slides (easier to read) – Cam.Davidson.Pilon Apr 13 '13 at 2:06

To expand on Zach's answer (+1), if you use the LASSO method in linear regression, you are trying to minimize the sum a quadratic function and an absolute value function, ie:

$$\min_{\beta} \; \; (Y-X\beta)^{T}(Y-X\beta) + \sum_i |\beta_i| $$

The first part is quadratic in $\beta$ (gold below), and the second is a square shaped curve (green below). The black line is the line of intersection. The LASSO objective function

The minimum lies on the curve of intersection, plotted here with the contour curves of the quadratic and square-shaped curve:

Contour curves of LASSO

You can see the minimum is on one of the axes, hence it has eliminated that variable from the regression.

You can check out my blog post on using $L1$ penalties for regression and variable selection (otherwise known as Lasso regularization).

share|improve this answer
(+1) but for the blog post, which is really good. It would be nice if you were to expand your answer here somewhat though, as this will increase the likelihood of the information remaining available. – richiemorrisroe Aug 21 '12 at 13:29

Whatever you do, it is worthwhile getting bootstrap confidence intervals on the ranks of importance of the predictors to show that you can really do this with your dataset. I am doubtful that any of the methods can reliably find the "true" predictors.

share|improve this answer

What is your prior belief on how many predictors are likely to be important? Is it likely that most of them have an exactly zero effect, or that everything affects the outcome, some variables only less than others?

And how is the health status related to the predictive task?

If you believe that only few variables are important, you may try the spike and slab prior (in the R's spikeSlabGAM package, for example), or L1. If you think all predictors affect the outcome, you may be out of luck.

And in general, all caveats related to causal inference from observational data apply.

share|improve this answer

I remember Lasso regression doesn't perform very well when $n \leq p$, but I'm not sure. I think in this case Elastic Net is more appropriate for variable selection.

share|improve this answer
This is true, more specifically when n<<p, see this original elastic net paper:… – Cam.Davidson.Pilon Aug 23 '12 at 15:10
When n < p, LASSO selects at most n variables. – miura Mar 7 '13 at 13:46

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.