# Detecting significant predictors out of 300 independent variables

In a dataset of two non-overlapping populations (patients & healthy, total $n=60$) I would like to find (out of $300$ independent variables) significant predictors for a continuous dependent variable. Correlation between predictors is present. I am interested in finding out if any of the predictors is related to the dependent variable "in reality" (rather than predicting the dependent variable as exactly as possible). As I got overwhelmed with the numerous possible approaches, I would like to ask for which approach is most recommended.

• From my understanding stepwise inclusion or exclusion of predictors, is not recommended

• E.g. run a linear regression separately for every predictor and correct p-values for multiple comparison using FDR (probably very conservative?)

• Principle-component regression: difficult to interpret as I won't be able to tell about the predictive power of individual predictors but only of the components.

• any other suggestions?

-
I've heard of people using L1 regularized regression to do this type of things. But I do not know enough to write a proper answer... – King Aug 21 '12 at 12:45
In order to give the best recommendations, it would help us to know how you will proceed after identifying "significant predictors." Are you trying to predict the outcome as exactly as possible; find a parsimonious way to predict it (e.g., using a set of up to k predictors that will efficiently do so; explain what causes the outcome "in reality"; or something else? Also, how big is your data set? – rolando2 Aug 21 '12 at 15:40
@rolando: thanks for the comment! I updated the question: my total number of observations is n=60 subjects. My aim is not to predict the dependent variable as exactly as possible but rather to explain what causes the outcome "in reality" (= hope to find relationsship between variables that could be confirmed in later studies/datasets) – jokel Aug 21 '12 at 21:16
I also posted a follow-up question including some dummy data. I would be very thankful for all hints. stats.stackexchange.com/questions/34859/… – jokel Aug 22 '12 at 15:40

I would recommend trying a glm with lasso regularization. This adds a penalty to the model for number of variables, and as you increase the penalty, the number of variables in the model will decrease.

You should use cross-validation to select the value of the penalty parameter. If you have R, I suggest using the glmnet package. Use alpha=1 for lasso regression, and alpha=0 for ridge regression. Setting a value between 0 and 1 will use a combination of lasso and ridge penalties, also know as the elastic net.

-
I agree with Zach. David Cassell and I wrote a paper about this, concentrating on SAS but not entirely. It is Stopping Stepwise. – Peter Flom Aug 21 '12 at 13:24
I think it's 0 for ridge and 1 for lasso – King Aug 21 '12 at 14:01
@King you are correct, I got it backwards. I'll update my answer. – Zach Aug 21 '12 at 17:49
@Zach: Thanks for the hints. Is there a way to obtain some test-statistic that would allow me to judge the significance of single predictors. In the end I would like to be able to say "predictor X is significantly related to dependent variable Y". – jokel Aug 22 '12 at 10:22
@miura Recently introduced was a test statistics for just that, by the original Lasso authors too: paper and slides (easier to read) – Cam.Davidson.Pilon Apr 13 at 2:06

To expand on Zach's answer (+1), if you use the LASSO method in linear regression, you are trying to minimize the sum a quadratic function and an absolute value function, ie:

$$\min_{\beta} \; \; (Y-X\beta)^{T}(Y-X\beta) + \sum_i |\beta_i|$$

The first part is quadratic in $\beta$ (gold below), and the second is a square shaped curve (green below). The black line is the line of intersection.

The minimum lies on the curve of intersection, plotted here with the contour curves of the quadratic and square-shaped curve:

You can see the minimum is on one of the axes, hence it has eliminated that variable from the regression.

You can check out my blog post on using $L1$ penalties for regression and variable selection (otherwise known as Lasso regularization).

-
(+1) but for the blog post, which is really good. It would be nice if you were to expand your answer here somewhat though, as this will increase the likelihood of the information remaining available. – richiemorrisroe Aug 21 '12 at 13:29

I would suggest a Generalized additive model, which is likelihood based and highly interpretable, because of its additive nature under continuous variable assumptions. The model fits a separate nonlinear smooth function to each predictor/feature in such a way that the sum of these functions gives the predicted response, i.e: $\hat{Y}=\sum_{i}f_i(X_i)$. The smoother a variable's function is, the less it is important. Lesser the smoothness, more the importance. Plots of these smooths can be made and are highly interpretable apart from looking at likelihood values of each fit. Also, it allows for choosing distributional assumptions other than gaussian, from the exponential family. Allows for non-stationary modeling, and the fits being non-linear allows for a wide-range of predictive applications.

-
 On the other hand, lasso deals with sparse model-selection but does not allow for modeling with higher complexity of functions, as it is a linear model and not a non-linear fit. – Praneeth Vepakomma Aug 23 '12 at 12:59 A GAM would have even more degrees of freedom than the linear model, and here already n=60, p=300. Sorry, the problem here is to choose a "shrinkage" method, not the functional form. – scellus Mar 7 at 15:08

I remember Lasso regression doesn't perform very well when n <= p, but I'm not sure. I think in this case Elastic Net is more appropriate for the variable selection.

-
This is true, more specifically when n<<p, see this original elastic net paper: stanford.edu/~hastie/Papers/… – Cam.Davidson.Pilon Aug 23 '12 at 15:10
When n < p, LASSO selects at most n variables. – miura Mar 7 at 13:46

What is your prior belief on how many predictors are likely to be important? Is it likely that most of them have an exactly zero effect, or that everything affects the outcome, some variables only less than others?

And how is the health status related to the predictive task?

If you believe that only few variables are important, you may try the spike and slab prior (in the R's spikeSlabGAM package, for example), or L1. If you think all predictors affect the outcome, you may be out of luck.

And in general, all caveats related to causal inference from observational data apply.

-