Different regularization parameter per parameter

I have never seen a regularization parameter (usually lambda or alpha) be different for each parameter. People consider different regularization parameters, but I believe they penalize all the parameters with equal strength.

Consider a linear regression with intercept and 2 predictors.

A suggestion for the regularization: instead of $\lambda \sum B_i^2$ consider $\sum(\lambda_i * B_i^2)$, from 1 to n, where i is the i-th parameter.

While generally a single $\lambda$ would be applied on all coefficients, we might have a vector of lambda, one for each coefficient (except the intercept). For $B_1$, $\lambda$ might be 5, while $\lambda$ for $B_2$ would be 10.

Have people used different regularization parameters for different fitted parameters, and are there any reasons to do so? When would be such a case?

One could imagine that by theory one would rather shrink one parameter more than another.

• Is that not just the same question as whether/how to standardize the predictors? – Scortchi Dec 6 '13 at 10:24
• @Scortchi I don't see that in it? Even when all predictors are on the same exact scale, couldn't it be that the penalties would be defined differently for each parameter? – PascalVKooten Dec 6 '13 at 10:33
• Could be; what I mean is that I think choosing appropriate scales for each parameter amounts to the same thing. (I think - haven't worked it through.) – Scortchi Dec 6 '13 at 10:43
• Interesting thought. But let me put it like this: no matter what scale you put variables on, penalties for each parameter could be, relatively and absolutely speaking, different. – PascalVKooten Dec 6 '13 at 11:34
• Following up on that thought, I found: Bien et al (2012), "A lasso for hierarchical testing of interactions", which uses two different $\lambda$s just like you said. – Scortchi Dec 6 '13 at 12:02