# Howe to perform ridge regression only on a subset of the variables

I am trying to code some algorithm that performs ridge-regression with penalty parameter $$\lambda$$ on all features except for a specific subset.

Let $$\mathbf{X}$$ be the $$n \times p$$ matrix for $$n$$ samples and $$p$$ features, $$\mathbb{y}$$ is our vector with the dependent variable for each sample.

If we applied ridge regression with penalization to ALL features, we would have our fit:

$$\mathbf{\hat{y} = X \hat{\beta} = X(X^T X+\lambda I)^{-1}Xy}$$

Nevertheless, what are you supposed to do if you don't want to penalize, say the second feature $$p = 2$$? Do we just instead of having a constant $$\lambda$$, let $$\lambda^*$$ be the penalty for the variables we want to penalize, and then

$$\lambda_i = \begin{cases}\lambda^*, & i \neq 2 \\ 0, & i = 2\end{cases}$$?

Or which way is it supposed to work?

## 1 Answer

Yes, it is valid to apply different penalties to different coefficients. From the documentation for glmnet:

penalty.factor: Separate penalty factors can be applied to each coefficient. This is a number that multiplies ‘lambda’ to allow differential shrinkage. Can be 0 for some variables, which implies no shrinkage, and that variable is always included in the model. Default is 1 for all variables (and implicitly infinity for variables listed in ‘exclude’). Note: the penalty factors are internally rescaled to sum to nvars, and the lambda sequence will reflect this change.

Formally you could say this uses $$\lambda^*$$ where $$\lambda^* = a\lambda$$ and $$a$$ is a vector of penalty factors.