11
$\begingroup$

When I determine my lambda through cross-validation, all coefficients become zero. But I have some hints from the literature that some of the predictors should definitely affect the outcome. Is it rubbish to arbitrarily choose lambda so that there is just as much sparsity as one desires?

I want to select the top 10 or so predictors out of 135 for a cox model and effect sizes unfortunately are small.

$\endgroup$
3
  • 6
    $\begingroup$ Sounds like you should be using an informative prior, as you have non-data based information. $\endgroup$ Jul 6, 2012 at 12:30
  • $\begingroup$ Deep down I feel like that would be correct, unfortunately I completely lack the statistical prowess to even now where to start doing this. $\endgroup$
    – miura
    Jul 16, 2012 at 6:46
  • 1
    $\begingroup$ You appear to confuse two different things: (1) If the literature tells you to use specific predictors, then include them in all models. (2) Instead you seem to re-interpret this as indicating you should select a certain number out of many predictors, regardless whether they include the specific ones mentioned in the literature. Could you clarify what you're actually trying to accomplish? $\endgroup$
    – whuber
    Jul 26, 2017 at 18:21

3 Answers 3

4
$\begingroup$

If you want to have at least a definite number of predictors with some range of values defined by the literature, why choose the pure-LASSO approach to begin with? As @probabilityislogic suggested, you should be using some informative priors on those variables where you have some knowledge about. If you want to retain some of the LASSO properties for the rest of the predictors, maybe you could use a prior with a double exponential distribution for each other input, i.e., use a density of the form $$p(\beta_i)=\frac{\lambda}{2}\text{exp}\left(-\lambda|\beta_i|\right),$$ where $\lambda$ is the lagrange multiplier corresponding to the pure-LASSO solution. This last statement comes from the fact that, in the absense of the variables with the informative priors, this is another way of deriving the LASSO (by maximizing the posterior mode given normality assumptions for the residuals).

$\endgroup$
3
$\begingroup$

There exists a nice way to perform LASSO but use a fixed number of predictors. It is Least angle regression (LAR or LARS) described in Efron's paper. During iterative procedure it creates a number of linear models, each new one has one more predictor, so you can select one with desired number of predictors.

Another way is $l_1$ or $l_2$ regularization. As mentioned by Nestor using appropriate priors you can incorporate prior knowledge into the model. So called relevance vector machine by Tipping can be useful.

$\endgroup$
3
  • 3
    $\begingroup$ While LARS and the lasso are closely related, for a fixed number of predictors, they may not even include the same variables. One could pick a penalty value for the lasso that gives the desired number of predictors, but the choice in neither case will be unique! Hence the OP has not yet provided a well-defined procedure, which is part of the problem. For LARS, there is the nice benefit that the penalty values yielding a certain number of predictors forms an interval, so picking an endpoint (which one?) or midpoint or some other criterion is somewhat easier. $\endgroup$
    – cardinal
    Jul 6, 2012 at 16:35
  • 1
    $\begingroup$ Yes, it is true that LARS and LASSO aren't identical, but a simple modification of LARS suggested by authors in the original article can be introduced to obtain LASSO solutions using LARS-based technique. $\endgroup$ Jul 6, 2012 at 16:59
  • $\begingroup$ Yes, Alexey, this is true. I guess my comment revolves around why moving to LARS in the first place. One could usually just as easily choose a value of the penalty parameter for the lasso that yields the desired number of predictors. The main point left unaddressed is how one should go about making a unique selection and the consequences that might have in the OP's case. :) $\endgroup$
    – cardinal
    Jul 6, 2012 at 17:13
2
$\begingroup$

No, that is not defensible. The great hurdle that model selection procedures are designed to overcome is that the the cardinality of the true support $\left| S^* \right| = \left| \left\{ j : \beta^*_j \neq 0 \right\} \right|$ is unknown. (Here we have that $\beta^*$ is the "true" coefficient.) Because $|S^*|$ is unknown, a model selection procedure has to exhaustively search over all $2^p$ possible models; however, if we did know $|S^*|$, we could just check the ${p \choose |S^*|}$ models, which is far fewer.

The theory of the lasso relies on the regularization parameter $\lambda$ being sufficiently large so as to make the selected model sufficiently sparse. It could be that your 10 features are too many or too few, since it isn't trivial to turn a lower bound on $\lambda$ into an upper bound on $|S^*|$.

Let $\hat\beta$ be our data-driven estimate for $\beta^*$, and put $\hat{S} = \{j \, : \, \hat\beta_j \neq 0 \}$. Then, perhaps you're trying to ensure that $S^* \subseteq \hat{S}$ so that you've recovered at least the relevant features? Or maybe you're trying to establish that $\hat{S} \subseteq S^*$ so that you know that features you've found are all worthwhile? In these cases, your procedure would be more justified if you had prior information on the relative sizes of $S^*$.

Also, note, you can leave some coefficients unpenalized when performing lasso in, for instance, glmnet.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.