2
$\begingroup$

I have never seen a regularization parameter (usually lambda or alpha) be different for each parameter. People consider different regularization parameters, but I believe they penalize all the parameters with equal strength.

Consider a linear regression with intercept and 2 predictors.

A suggestion for the regularization: instead of $\lambda \sum B_i^2$ consider $\sum(\lambda_i * B_i^2)$, from 1 to n, where i is the i-th parameter.

While generally a single $\lambda$ would be applied on all coefficients, we might have a vector of lambda, one for each coefficient (except the intercept). For $B_1$, $\lambda$ might be 5, while $\lambda$ for $B_2$ would be 10.

Have people used different regularization parameters for different fitted parameters, and are there any reasons to do so? When would be such a case?

One could imagine that by theory one would rather shrink one parameter more than another.

$\endgroup$
  • $\begingroup$ Is that not just the same question as whether/how to standardize the predictors? $\endgroup$ – Scortchi Dec 6 '13 at 10:24
  • $\begingroup$ @Scortchi I don't see that in it? Even when all predictors are on the same exact scale, couldn't it be that the penalties would be defined differently for each parameter? $\endgroup$ – PascalVKooten Dec 6 '13 at 10:33
  • $\begingroup$ Could be; what I mean is that I think choosing appropriate scales for each parameter amounts to the same thing. (I think - haven't worked it through.) $\endgroup$ – Scortchi Dec 6 '13 at 10:43
  • $\begingroup$ Interesting thought. But let me put it like this: no matter what scale you put variables on, penalties for each parameter could be, relatively and absolutely speaking, different. $\endgroup$ – PascalVKooten Dec 6 '13 at 11:34
  • 1
    $\begingroup$ Following up on that thought, I found: Bien et al (2012), "A lasso for hierarchical testing of interactions", which uses two different $\lambda$s just like you said. $\endgroup$ – Scortchi Dec 6 '13 at 12:02
6
$\begingroup$

Yes, it had been tried (including by myself - I tried it with neural nets, with rather mixed success). The Relevance Vector Machine (RVM) does pretty much exactly that, and the regularisation parameters are tuned by maximising the marginal likelihood. The advantage of this is that it leads to a sparse model where uninformative attributes end up with large regularisation parameters and hence small weights. The problem with this approach lies in the tuning of the regularisation parameters, which tends to result in over-fitting the model selection criterion (whether Bayesian or cross-validation based), simply because there are many degrees of freedom introduced by having many hyper-parameters to tune.

$\endgroup$
0
$\begingroup$

Adaptive Lasso (H.Zou, JASA 2006, Vol. 101, No. 476) achieves consistency in parameter estimates by using individual lambda for each variable. Lambda values are tuned based on OLS solution (which unfortunately is not available in many practical cases where Lasso is used).

$\endgroup$
0
$\begingroup$

If you want to/are able to go nonparametric, this is implemented in the mgcv package, which implements penalized splines. If you use the option select=TRUE, the optimizer that selects smoothness penalties also adds a penalty term to the "main effect" of each smooth term, in addition to the penalty used for smoothness selection. It doesn't however implement this for the parametric part of the model, and it is computationally intensive.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.