1
$\begingroup$

I am trying to code some algorithm that performs ridge-regression with penalty parameter $\lambda$ on all features except for a specific subset.

Let $\mathbf{X}$ be the $n \times p$ matrix for $n$ samples and $p$ features, $\mathbb{y}$ is our vector with the dependent variable for each sample.

If we applied ridge regression with penalization to ALL features, we would have our fit:

$$\mathbf{\hat{y} = X \hat{\beta} = X(X^T X+\lambda I)^{-1}Xy}$$

Nevertheless, what are you supposed to do if you don't want to penalize, say the second feature $p = 2$? Do we just instead of having a constant $\lambda$, let $\lambda^*$ be the penalty for the variables we want to penalize, and then

$$\lambda_i = \begin{cases}\lambda^*, & i \neq 2 \\ 0, & i = 2\end{cases}$$?

Or which way is it supposed to work?

$\endgroup$
1
$\begingroup$

Yes, it is valid to apply different penalties to different coefficients. From the documentation for glmnet:

penalty.factor: Separate penalty factors can be applied to each coefficient. This is a number that multiplies ‘lambda’ to allow differential shrinkage. Can be 0 for some variables, which implies no shrinkage, and that variable is always included in the model. Default is 1 for all variables (and implicitly infinity for variables listed in ‘exclude’). Note: the penalty factors are internally rescaled to sum to nvars, and the lambda sequence will reflect this change.

Formally you could say this uses $\lambda^*$ where $\lambda^* = a\lambda$ and $a$ is a vector of penalty factors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.