2
$\begingroup$

I know for ridge regression, we add a penalty term with a fixed constant λ for all β the as tuning parameter. Just wonder why choosing a fixed λ instead of varying its value for different β?

Like this:

enter image description here

$\endgroup$
2
$\begingroup$

If you varied the penalty for every $\beta$ the effective number of parameters would be $2\times p$ and you would have overfitting once again. Essentially the model would be non-identifiable. Penalization works by borrowing information or tilting estimates towards a preconceived bias in a systematic way. If you don't connect the $\beta$s in some way you are not borrowing information. What would work would be to group the $X$s using prior knowledge about the likely importance of them, and to give a different $\lambda$ to each group, assuming every group has $> 1$ predictor in it. The pre-grouping would not be allowed to use $Y$.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.