Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 0 down vote favorite

Suppose I want to answer the following question: Rank the explanatory variables from most important to least important.

I want to do the following: Pick lambda from largest to small, and report in a table : when lambda is 100, 80, 60, 40, 20, 10, the explanatory variables are such and such and their loading are such and such.

1.Is it legal to do this? Each lambda represent a Laplace prior, reporting multiple lambda values would be equivalent to have multiple priors. It seems weird we can have multiple priors. 2.If it is not legal, what method should I use to answer my original question.

share|improve this question
You could choose lambda by cross-validation. There is a function in the R package glmnet which does this for you. – dave Jan 15 at 3:25
1  
I was going to answer as per dave's comment, but cross-validation assumes you are building a predictive model, as opposed to solely ranking the variables. In this case, AdamO's answer seems appropriate. If on the other hand you are ranking variables as the first step to building a model, then using cross-validation to find the optimal lambda value becomes appropriate once more. It depends on what your ultimate goal is. – Bogdanovist Jan 15 at 5:02
What you describe in your question is exactly what a plot of the LASSO regularization path shows. Just google it. This kind of plot is implemented in package glmnet (plot.glmnet) or penalized (plotpath). With both these packages you can also easily compute the coefficient profile at desired values of lambda to summarize in a table. – miura Jan 15 at 8:43
Can you describe what is cross validation? – Zwingli Tannenbaum Jan 16 at 23:09

1 Answer

up vote 1 down vote

I don't intuitively feel this answers any question we'd be interested in. One uses the LASSO to obtain model based predictions with sparse estimates of model coefficients when $p$ is much larger than $n$. When used for feature selection, the problem is that $X_1$ may be in the "1-variable model" (whatever value of $\lambda$ that leads to a single covariate), whereas $X_{20}$ and $X_{151}$ might be in the "2-variable model" once $\lambda$ has been sufficiently decreased. $X_1$ also may not show up in any subsequent model and have very poor reliability for prediction. So we can't say $X_1$ is the "most important factor".

I think another important caveat in your problem description is that the actual continuous values of $\lambda$ are meaningless. Fitting models over a large sequence of possible values for $\lambda$ can be time consuming, despite that the LASSO is computationally efficient.

To rank factors sequentially according to their "importance" (which I would assume to mean as predictors in some type of linear model with continuous outcomes), I would look at a forward stepwise model selection process. This has an implicit comparison to nested models using formal inference. There are other criteria rather than $p-values$ that one can use for this process, though, including information criteria which penalize incremental benefits in bigger models due to the addition of parameters.

share|improve this answer
In the $p >> n$ case, LASSO selects at most $n$ of the $p$ predictors. Depending on your goal, this might be undesirable. In stepwise regression, usage of AIC or BIC minimization as selection criterion has a one-to-one correspondence to particular p-values, depending on the quantiles of the chi-squared distribution. For example, AIC minimization corresponds to a p-value of 0.157 for variables with one degree of freedom. – miura Jan 15 at 8:38
The attractive feature of stepwise model selection, despite its obvious shortcomings (namely the correspondence to significance based variable selection), is that gives consistent sequential rankings of variables, so if "age" is the primary risk factor for CHD, the next variable in the model "sex" will indicate the second more important factor after age. This makes reporting much easier. – AdamO Jan 18 at 20:44

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.