# Why does fitting the hyperparameter of Ridge regression at the same time as the model parameters does not lead to a vanishing hyperparameter?

I have been simulating some quadratic data with some noise (constant for all points) into it. I am fitting those data with a polynomial fit with Ridge regression. To find the best hyperparameter, I am using cross-validation and I get a great model afterwards. My error function is: $$error=\sum(\hat y(w) -y)^2 + \lambda||w||^2$$

My issue is, when I tried to fit $$\lambda$$ just for the fun of it, I do not get a value of 0. I was expecting a value of 0 because in my mind, since the Ridge factor acts as a a constraint, it forces the solution to go away from the least-square one, so the least-square factor in my error function gets bigger, and we also have an additional term to the error that makes the overall error bigger. Hence, I was expecting $$\lambda$$ to go to 0 to recover the least-square solution, but this is not what I get at all. Here are some plots to illustrates.

This is when I fit my 10 degree polynomial to my dataset without the Ridge factor. The grey region is the 1 sigma range (I am doing Bayesian analysis). The red line is the real model.

This is when I use Ridge regression and use the $$\lambda$$ obtained through cross-validation.

This is when I fit $$\lambda$$ at the same time as my polynomial coefficients. The $$\lambda$$ retrieved by cross-validation and Fitting the $$\lambda$$ are very similar, as you can see from the graphs

We can see that the two last plot are very similar. I'm not interesting in generalizing my predictions away from this region, I am only interested in retrieving the model that is closest to the real one. To quantify that, I use the L2 norm, $$\int (\hat f-f)^2dx$$ where $$\hat f$$ is the retrieved model and $$f$$ is the real model. The model with the lowest difference to the real one is obtained from cross-validation as expected, but the model with the $$\lambda$$ fitted is really close too.

As explained above, I dot not understand why $$\lambda$$ does not go to 0 and that I don't retrieve the solution from the first graph while fitting $$\lambda$$.

One of the possible explanation is from the exact error function that I use. As explained above, I do bayesian analysis, and so my "error" function described above is not exactly the one that I am using. For those familiar, the functional form of my Log posterior is:

$$log Lp=-\frac{1}{2\sigma^2}\sum(\hat y-y)^2-\log(\sigma \sqrt{2\pi}) -\frac{1}{2\sigma_s^2}||w||^2 -\log(\sigma_s \sqrt{2\pi})$$

Where $$\sigma$$ is my gaussian noise to the data. It is fitted during the bayesian analysis. $$\sigma_s$$ is the standard deviation of the gaussian prior centered at 0 on the weights $$w$$. $$\frac{1}{2\sigma_s^2} =\lambda$$ is the usual Ridge factor.

I could see some weird interplay between the two log terms while fitting my data that would explain why $$\frac{1}{2\sigma_s^2}$$ does not go to 0, but I don't really buy it.

For reference, I am doing the regression by treating each power of x as a feature and normalizing each of those to avoid scaling issues when doing Ridge.

Anyway, thanks for the help!

• So what did you do more precisely? If I am not mistaken you wrote the expression for $log P(w|Y_{train},X_{train},\sigma,\sigma_s)$, where the prior $P(w|\sigma_s)=N(0,\sigma_s^2)$. From here what did you do to "optimize" $\sigma_s$ ? Feb 10, 2021 at 20:55
• When I "fit" $\sigma_s$, I perform a MCMC retrieval on the full parameter space+ $\sigma_s$. I then look at the marginalized distribution for $\sigma_s$ and to my suprise it wasn't centered at 0. Feb 10, 2021 at 21:51
• It would be useful I think to have more details to answer. This is what I guess from what you said. You first found an expression for $p(\sigma_s,w,\sigma|X_{tr},Y_{tr})$ and than you used this to evaluate (analytically) the conditional probabilities needed to perform Gibbs sampling ( en.wikipedia.org/wiki/Gibbs_sampling ) ? Than you did an histogram of the marginal $p(\sigma_s|X_{tr},Y_{tr})$ from this experiment. Is this correct ? Or what was the MCMC flavor that you used ( MCMC is not a method, but rather a class of method, I just guessed here that you are using Gibbs sampling ) ? Feb 12, 2021 at 8:45
• In my opinion, it would help people to answer properly your question, if you added these details ;) Feb 12, 2021 at 8:51