# Is the error linear in terms of $\lambda$ in ridge regression?

I have a data set of $(X_n, t_n)$ that I am doing a ridge regression on.

I am also doing $10$ fold cross validation to fine tune the value of $\lambda$ for values from $0$ to $4.0$ in $0.1$ step increments. However, after I did the cross validation and plotted the values of $\lambda$ against the ridge error function: $\sum_{n=1}^N(t_n-w^TX_n)^2 + \frac{\lambda}{2}||w||^2$, I got this curve:

I plotted a scatter plot of the points ($X_n$ is in $\mathbb{R}^2$) and the points looked VERY close to all lying on the same plane:

Could it really be that this is how the error varies as $\lambda$ varies or is there something wrong in my implementation?

• (1) The first plot clearly is not linear, although it may be approximately so. (2) What does the second plot show? Without labels on the axes or a description, it's not giving us much information. – whuber Jan 12 '18 at 21:40
• @whuber, (2) on the the bottom plane (20x20), it represents the values of both features in the vector $X_i$ and the $z$ value corresponds to $t_n$. – AspiringMat Jan 12 '18 at 21:46

I found out what I was doing wrong. After I calculate the coefficients of the regression $w$, I was calculating the MSE of that prediction wrongly, meaning I was adding the term $+\frac{\lambda}{2}||w||^2$ while calculating the MSE in the end, when this term is only needed when calculating $w$ only!

I simply changed the MSE calculation to $\sum_{n=1}^N (t_n-w^TX_n)^2$ and I got this curve which makes more sense: