Howe to perform ridge regression only on a subset of the variables

Question

I am trying to code some algorithm that performs ridge-regression with penalty parameter $\lambda$ on all features except for a specific subset.

Let $\mathbf{X}$ be the $n \times p$ matrix for $n$ samples and $p$ features, $\mathbb{y}$ is our vector with the dependent variable for each sample.

If we applied ridge regression with penalization to ALL features, we would have our fit:

$$\mathbf{\hat{y} = X \hat{\beta} = X(X^T X+\lambda I)^{-1}Xy}$$

Nevertheless, what are you supposed to do if you don't want to penalize, say the second feature $p = 2$? Do we just instead of having a constant $\lambda$, let $\lambda^*$ be the penalty for the variables we want to penalize, and then

$$\lambda_i = \begin{cases}\lambda^*, & i \neq 2 \\ 0, & i = 2\end{cases}$$?

Or which way is it supposed to work?

Answer 1

Yes, it is valid to apply different penalties to different coefficients. From the documentation for glmnet:

penalty.factor: Separate penalty factors can be applied to each coefficient. This is a number that multiplies ‘lambda’ to allow differential shrinkage. Can be 0 for some variables, which implies no shrinkage, and that variable is always included in the model. Default is 1 for all variables (and implicitly infinity for variables listed in ‘exclude’). Note: the penalty factors are internally rescaled to sum to nvars, and the lambda sequence will reflect this change.

Formally you could say this uses $\lambda^*$ where $\lambda^* = a\lambda$ and $a$ is a vector of penalty factors.