32
$\begingroup$

In a dataset of two non-overlapping populations (patients & healthy, total $n=60$) I would like to find (out of $300$ independent variables) significant predictors for a continuous dependent variable. Correlation between predictors is present. I am interested in finding out if any of the predictors are related to the dependent variable "in reality" (rather than predicting the dependent variable as exactly as possible). As I got overwhelmed with the numerous possible approaches, I would like to ask for which approach is most recommended.

  • From my understanding stepwise inclusion or exclusion of predictors is not recommended

  • E.g. run a linear regression separately for every predictor and correct p-values for multiple comparison using FDR (probably very conservative?)

  • Principal-component regression: difficult to interpret as I won't be able to tell about the predictive power of individual predictors but only about the components.

  • any other suggestions?

Improve this question
$\endgroup$
4
  • $\begingroup$ I've heard of people using L1 regularized regression to do this type of things. But I do not know enough to write a proper answer... $\endgroup$
    – King
    Aug 21, 2012 at 12:45
  • 2
    $\begingroup$ In order to give the best recommendations, it would help us to know how you will proceed after identifying "significant predictors." Are you trying to predict the outcome as exactly as possible; find a parsimonious way to predict it (e.g., using a set of up to k predictors that will efficiently do so; explain what causes the outcome "in reality"; or something else? Also, how big is your data set? $\endgroup$
    – rolando2
    Aug 21, 2012 at 15:40
  • $\begingroup$ @rolando: thanks for the comment! I updated the question: my total number of observations is n=60 subjects. My aim is not to predict the dependent variable as exactly as possible but rather to explain what causes the outcome "in reality" (= hope to find relationsship between variables that could be confirmed in later studies/datasets) $\endgroup$
    – jokel
    Aug 21, 2012 at 21:16
  • $\begingroup$ I also posted a follow-up question including some dummy data. I would be very thankful for all hints. stats.stackexchange.com/questions/34859/… $\endgroup$
    – jokel
    Aug 22, 2012 at 15:40

5 Answers 5

Reset to default
31
$\begingroup$

I would recommend trying a glm with lasso regularization. This adds a penalty to the model for number of variables, and as you increase the penalty, the number of variables in the model will decrease.

You should use cross-validation to select the value of the penalty parameter. If you have R, I suggest using the glmnet package. Use alpha=1 for lasso regression, and alpha=0 for ridge regression. Setting a value between 0 and 1 will use a combination of lasso and ridge penalties, also know as the elastic net.

Improve this answer
$\endgroup$
9
  • 5
    $\begingroup$ I agree with Zach. David Cassell and I wrote a paper about this, concentrating on SAS but not entirely. It is Stopping Stepwise. $\endgroup$
    – Peter Flom
    Aug 21, 2012 at 13:24
  • 1
    $\begingroup$ I think it's 0 for ridge and 1 for lasso $\endgroup$
    – King
    Aug 21, 2012 at 14:01
  • 1
    $\begingroup$ @Zach: Thanks for the hints. Is there a way to obtain some test-statistic that would allow me to judge the significance of single predictors. In the end I would like to be able to say "predictor X is significantly related to dependent variable Y". $\endgroup$
    – jokel
    Aug 22, 2012 at 10:22
  • 2
    $\begingroup$ Regarding CIs, from the manual of another R package implementing the LASSO (cran.r-project.org/web/packages/penalized/vignettes/…, page 18): "It is a very natural question to ask for standard errors of regression coefficients or other estimated quantities. In principle such standard errors can easily be calculated, e.g. using the bootstrap. Still, this package deliberately does not provide them. The reason for this is that standard errors are not very meaningful for strongly biased estimates such as arise from penalized estimation methods." $\endgroup$
    – miura
    Sep 19, 2012 at 9:01
  • 2
    $\begingroup$ @miura Recently introduced was a test statistics for just that, by the original Lasso authors too: paper and slides (easier to read) $\endgroup$
    – Cam.Davidson.Pilon
    Apr 13, 2013 at 2:06
24
$\begingroup$

To expand on Zach's answer (+1), if you use the LASSO method in linear regression, you are trying to minimize the sum a quadratic function and an absolute value function, ie:

$$\min_{\beta} \; \; (Y-X\beta)^{T}(Y-X\beta) + \sum_i |\beta_i| $$

The first part is quadratic in $\beta$ (gold below), and the second is a square shaped curve (green below). The black line is the line of intersection. The LASSO objective function

The minimum lies on the curve of intersection, plotted here with the contour curves of the quadratic and square-shaped curve:

Contour curves of LASSO

You can see the minimum is on one of the axes, hence it has eliminated that variable from the regression.

You can check out my blog post on using $L1$ penalties for regression and variable selection (otherwise known as Lasso regularization).

Improve this answer
$\endgroup$
1
  • 8
    $\begingroup$ (+1) but for the blog post, which is really good. It would be nice if you were to expand your answer here somewhat though, as this will increase the likelihood of the information remaining available. $\endgroup$
    – richiemorrisroe
    Aug 21, 2012 at 13:29
2
$\begingroup$

What is your prior belief on how many predictors are likely to be important? Is it likely that most of them have an exactly zero effect, or that everything affects the outcome, some variables only less than others?

And how is the health status related to the predictive task?

If you believe that only few variables are important, you may try the spike and slab prior (in the R's spikeSlabGAM package, for example), or L1. If you think all predictors affect the outcome, you may be out of luck.

And in general, all caveats related to causal inference from observational data apply.

Improve this answer
$\endgroup$
2
$\begingroup$

Whatever you do, it is worthwhile getting bootstrap confidence intervals on the ranks of importance of the predictors to show that you can really do this with your dataset. I am doubtful that any of the methods can reliably find the "true" predictors.

Improve this answer
$\endgroup$
2
$\begingroup$

I remember Lasso regression doesn't perform very well when $n \leq p$, but I'm not sure. I think in this case Elastic Net is more appropriate for variable selection.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is true, more specifically when n<<p, see this original elastic net paper: stanford.edu/~hastie/Papers/… $\endgroup$
    – Cam.Davidson.Pilon
    Aug 23, 2012 at 15:10
  • 1
    $\begingroup$ When n < p, LASSO selects at most n variables. $\endgroup$
    – miura
    Mar 7, 2013 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.