# Different regularization parameter per parameter

I have never seen a regularization parameter (usually lambda or alpha) be different for each parameter. People consider different regularization parameters, but I believe they penalize all the parameters with equal strength.

Consider a linear regression with intercept and 2 predictors.

A suggestion for the regularization: instead of $\lambda \sum B_i^2$ consider $\sum(\lambda_i * B_i^2)$, from 1 to n, where i is the i-th parameter.

While generally a single $\lambda$ would be applied on all coefficients, we might have a vector of lambda, one for each coefficient (except the intercept). For $B_1$, $\lambda$ might be 5, while $\lambda$ for $B_2$ would be 10.

Have people used different regularization parameters for different fitted parameters, and are there any reasons to do so? When would be such a case?

One could imagine that by theory one would rather shrink one parameter more than another.

• Is that not just the same question as whether/how to standardize the predictors? – Scortchi Dec 6 '13 at 10:24
• @Scortchi I don't see that in it? Even when all predictors are on the same exact scale, couldn't it be that the penalties would be defined differently for each parameter? – PascalVKooten Dec 6 '13 at 10:33
• Could be; what I mean is that I think choosing appropriate scales for each parameter amounts to the same thing. (I think - haven't worked it through.) – Scortchi Dec 6 '13 at 10:43
• Interesting thought. But let me put it like this: no matter what scale you put variables on, penalties for each parameter could be, relatively and absolutely speaking, different. – PascalVKooten Dec 6 '13 at 11:34
• Following up on that thought, I found: Bien et al (2012), "A lasso for hierarchical testing of interactions", which uses two different $\lambda$s just like you said. – Scortchi Dec 6 '13 at 12:02

If you want to/are able to go nonparametric, this is implemented in the mgcv package, which implements penalized splines. If you use the option select=TRUE, the optimizer that selects smoothness penalties also adds a penalty term to the "main effect" of each smooth term, in addition to the penalty used for smoothness selection. It doesn't however implement this for the parametric part of the model, and it is computationally intensive.