# Rewriting the Ridge Regression coefficients

In Ridge Regression we try to find the minimum of the following loss function:

$$\text{min}_w\mathcal{L}_{\lambda}(w,S)=\text{min}\lambda\|w\|^2+\sum^l_{i=1}(y_i-g(x_i))^2$$

Where:

• $$\lambda$$ is a positive number that defines the relative trade-off betweeen norm and loss
• $$\mathcal{L}$$ is the loss function
• $$w\in\mathbb{R}^n$$ is the vector of weights
• $$g(x_i)$$ is the predicted value of observation $$x_i$$

Taking the derivative of the cost function with respect to the parameters we obtain the equations (*)

$$X'Xw+\lambda w=(X'X+\lambda I_n)w=X'y$$

Where:

• $$I_n$$ is the $$n\times n$$ identity matrix
• $$X\in \mathbb{R}^{l\times n}$$ is the data matrix
• $$X'$$ is the transpose of $$X$$

The solution to the above equation is

$$w=(X'X+\lambda I_n)^{-1}X'y$$

Now, my book says that we can rewrite equations (*) in terms of $$w$$:

$$w=\lambda^{-1}X'(y-Xw)=X'\alpha$$

showing that $$w$$ can be written as a linear combination of the training points $$w=\sum^l_{i=1}\alpha_ix_i$$ with $$\alpha=\lambda^{-1}(y-Xw)$$

I have a hard time understanding how is $$w=\lambda^{-1}X'(y-Xw)$$ derived. Can someone show this algebraically?

• You should have a typo on equation (*), since it should be $X'Xw + \lambda w = (X'X +\lambda I_n)w= X'y$, then deriving the rest is quite straightforward Aug 17, 2021 at 12:37

$$X'y = X'Xw + \lambda w$$
$$X'y - X'Xw = \lambda w$$
$$X'(y - Xw) = \lambda w$$
$$w = \lambda^{-1}X'(y - Xw)$$
$$w = X'\alpha$$ with $$\alpha=\lambda^{-1}(y - Xw)$$