Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

In a dataset of two non-overlapping populations (patients & healthy, total $n=60$) I would like to find (out of $300$ independent variables) significant predictors for a continuous dependent variable. Correlation between predictors is present. I am interested in finding out if any of the predictors are related to the dependent variable "in reality" (rather than predicting the dependent variable as exactly as possible). As I got overwhelmed with the numerous possible approaches, I would like to ask for which approach is most recommended.

  • From my understanding stepwise inclusion or exclusion of predictors is not recommended

  • E.g. run a linear regression separately for every predictor and correct p-values for multiple comparison using FDR (probably very conservative?)

  • Principal-component regression: difficult to interpret as I won't be able to tell about the predictive power of individual predictors but only about the components.

  • any other suggestions?

share|improve this question
    
I've heard of people using L1 regularized regression to do this type of things. But I do not know enough to write a proper answer... –  King Aug 21 '12 at 12:45
1  
In order to give the best recommendations, it would help us to know how you will proceed after identifying "significant predictors." Are you trying to predict the outcome as exactly as possible; find a parsimonious way to predict it (e.g., using a set of up to k predictors that will efficiently do so; explain what causes the outcome "in reality"; or something else? Also, how big is your data set? –  rolando2 Aug 21 '12 at 15:40
    
@rolando: thanks for the comment! I updated the question: my total number of observations is n=60 subjects. My aim is not to predict the dependent variable as exactly as possible but rather to explain what causes the outcome "in reality" (= hope to find relationsship between variables that could be confirmed in later studies/datasets) –  jokel Aug 21 '12 at 21:16
    
I also posted a follow-up question including some dummy data. I would be very thankful for all hints. stats.stackexchange.com/questions/34859/… –  jokel Aug 22 '12 at 15:40
add comment

5 Answers

up vote 23 down vote accepted

I would recommend trying a glm with lasso regularization. This adds a penalty to the model for number of variables, and as you increase the penalty, the number of variables in the model will decrease.

You should use cross-validation to select the value of the penalty parameter. If you have R, I suggest using the glmnet package. Use alpha=1 for lasso regression, and alpha=0 for ridge regression. Setting a value between 0 and 1 will use a combination of lasso and ridge penalties, also know as the elastic net.

share|improve this answer
4  
I agree with Zach. David Cassell and I wrote a paper about this, concentrating on SAS but not entirely. It is Stopping Stepwise. –  Peter Flom Aug 21 '12 at 13:24
    
I think it's 0 for ridge and 1 for lasso –  King Aug 21 '12 at 14:01
    
@King you are correct, I got it backwards. I'll update my answer. –  Zach Aug 21 '12 at 17:49
1  
@Zach: Thanks for the hints. Is there a way to obtain some test-statistic that would allow me to judge the significance of single predictors. In the end I would like to be able to say "predictor X is significantly related to dependent variable Y". –  jokel Aug 22 '12 at 10:22
1  
@miura Recently introduced was a test statistics for just that, by the original Lasso authors too: paper and slides (easier to read) –  Cam.Davidson.Pilon Apr 13 '13 at 2:06
show 4 more comments

To expand on Zach's answer (+1), if you use the LASSO method in linear regression, you are trying to minimize the sum a quadratic function and an absolute value function, ie:

$$\min_{\beta} \; \; (Y-X\beta)^{T}(Y-X\beta) + \sum_i |\beta_i| $$

The first part is quadratic in $\beta$ (gold below), and the second is a square shaped curve (green below). The black line is the line of intersection. The LASSO objective function

The minimum lies on the curve of intersection, plotted here with the contour curves of the quadratic and square-shaped curve:

Contour curves of LASSO

You can see the minimum is on one of the axes, hence it has eliminated that variable from the regression.

You can check out my blog post on using $L1$ penalties for regression and variable selection (otherwise known as Lasso regularization).

share|improve this answer
7  
(+1) but for the blog post, which is really good. It would be nice if you were to expand your answer here somewhat though, as this will increase the likelihood of the information remaining available. –  richiemorrisroe Aug 21 '12 at 13:29
add comment

What is your prior belief on how many predictors are likely to be important? Is it likely that most of them have an exactly zero effect, or that everything affects the outcome, some variables only less than others?

And how is the health status related to the predictive task?

If you believe that only few variables are important, you may try the spike and slab prior (in the R's spikeSlabGAM package, for example), or L1. If you think all predictors affect the outcome, you may be out of luck.

And in general, all caveats related to causal inference from observational data apply.

share|improve this answer
add comment

Whatever you do, it is worthwhile getting bootstrap confidence intervals on the ranks of importance of the predictors to show that you can really do this with your dataset. I am doubtful that any of the methods can reliably find the "true" predictors.

share|improve this answer
add comment

I remember Lasso regression doesn't perform very well when $n \leq p$, but I'm not sure. I think in this case Elastic Net is more appropriate for variable selection.

share|improve this answer
    
This is true, more specifically when n<<p, see this original elastic net paper: stanford.edu/~hastie/Papers/… –  Cam.Davidson.Pilon Aug 23 '12 at 15:10
1  
When n < p, LASSO selects at most n variables. –  miura Mar 7 '13 at 13:46
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.