I have been simulating some quadratic data with some noise (constant for all points) into it. I am fitting those data with a polynomial fit with Ridge regression. To find the best hyperparameter, I am using cross-validation and I get a great model afterwards. My error function is: $error=\sum(\hat y(w) -y)^2 + \lambda||w||^2$
My issue is, when I tried to fit $\lambda$ just for the fun of it, I do not get a value of 0. I was expecting a value of 0 because in my mind, since the Ridge factor acts as a a constraint, it forces the solution to go away from the least-square one, so the least-square factor in my error function gets bigger, and we also have an additional term to the error that makes the overall error bigger. Hence, I was expecting $\lambda$ to go to 0 to recover the least-square solution, but this is not what I get at all. Here are some plots to illustrates.
This is when I fit my 10 degree polynomial to my dataset without the Ridge factor. The grey region is the 1 sigma range (I am doing Bayesian analysis). The red line is the real model.
This is when I use Ridge regression and use the $\lambda$ obtained through cross-validation.
This is when I fit $\lambda$ at the same time as my polynomial coefficients. The $\lambda$ retrieved by cross-validation and Fitting the $\lambda$ are very similar, as you can see from the graphs
We can see that the two last plot are very similar. I'm not interesting in generalizing my predictions away from this region, I am only interested in retrieving the model that is closest to the real one. To quantify that, I use the L2 norm, $\int (\hat f-f)^2dx$ where $\hat f$ is the retrieved model and $f$ is the real model. The model with the lowest difference to the real one is obtained from cross-validation as expected, but the model with the $\lambda$ fitted is really close too.
As explained above, I dot not understand why $\lambda$ does not go to 0 and that I don't retrieve the solution from the first graph while fitting $\lambda$.
One of the possible explanation is from the exact error function that I use. As explained above, I do bayesian analysis, and so my "error" function described above is not exactly the one that I am using. For those familiar, the functional form of my Log posterior is:
$log Lp=-\frac{1}{2\sigma^2}\sum(\hat y-y)^2-\log(\sigma \sqrt{2\pi}) -\frac{1}{2\sigma_s^2}||w||^2 -\log(\sigma_s \sqrt{2\pi}) $
Where $\sigma$ is my gaussian noise to the data. It is fitted during the bayesian analysis. $\sigma_s$ is the standard deviation of the gaussian prior centered at 0 on the weights $w$. $\frac{1}{2\sigma_s^2} =\lambda$ is the usual Ridge factor.
I could see some weird interplay between the two log terms while fitting my data that would explain why $\frac{1}{2\sigma_s^2}$ does not go to 0, but I don't really buy it.
For reference, I am doing the regression by treating each power of x as a feature and normalizing each of those to avoid scaling issues when doing Ridge.
Anyway, thanks for the help!
