4
$\begingroup$

I have the following question about the theoretical advantages vs. the empirical advantages of regularization (i.e. shrinkage).

As far as I understand, this is the general idea behind regularization: The "Bias-Variance Tradeoff" states that "simpler models" tend to be unable to capture complex patterns within the data and produce estimates with large biases - whereas more "complex models" tend to be able to better capture complex patterns in the data, but tend to generalize poorly to new data as they produce estimates that can greatly vary on unseen data. (note: "complexity" can be interpreted as a function of the number of parameters within the model)

As a result, regularization attempts to rectify this "Bias-Variance Tradeoff" by making complex models simpler, such that the complex models can still capture complex patterns but generalize better to unseen data. Regularization attempts to do this by "shrinking" model parameters towards 0. Doing so greatly reduces the effect these model parameters would have on the predictions, and thereby create "simplified complex model".

My Question: What proof is there that a regularized model has the (inherent) ability to overfit the data less than a non-regularized model?

I have seen some of the early proofs of the "Tikohonov Regularization Kernel" which outlines the "mathematical validity" of the Least Squares Regression Estimator when a regularization term is added to the optimization equation - but I have not seen any arguments that suggest a regularized model has the (inherent) ability to overfit the data less than a non-regularized model. Perhaps someone could argue that by virtue of the "No Free Lunch Theorem", there might in fact exist certain problems where a non-regularized model might perform better compared to a similar regularized model? I have seen countless empirical reports that display the benefits and advantages of using regularization - but I have not come across any "proofs" that suggest a "sparser model" (i.e. regularized) has the inherent ability to better perform than a similar non-regularized model.

Can someone please comment on this?

Thanks!

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Here is a thread discussing lower variance of ridge than OLS: stats.stackexchange.com/questions/245909/… These notes discuss (potential) MSE superiority: few.vu.nl/~wvanwie/Courses/HighdimensionalDataAnalysis/… $\endgroup$
    – Christoph Hanck
    Dec 21, 2021 at 12:55
  • 1
    $\begingroup$ Somewhat related: "Could applying ridge regression on small dataset improve predicting power?". $\endgroup$
    – Richard Hardy
    Dec 21, 2021 at 13:40
  • 1
    $\begingroup$ @stats_noob, I've noticed a trend where you ask questions that speak almost entirely in generalities and vague impressions. Another problem is that the use of terminology is either confused or incorrect. I think this speaks to a lack of familiarity with the fundamentals of the field. Please, take the time to read high-quality introductory textbooks and become familiar with the fundamentals. A good place to get started is here: stats.stackexchange.com/questions/12386/… $\endgroup$
    – Sycorax ♦
    Feb 21 at 3:53
  • $\begingroup$ @ Sycorax: thank you for your reply! I am trying to self teach myself these topics. Many textbooks tend to be too complicated for me to understand. I have a compromised immune system and can't really go outside because of the Corona Virus Pandemic - otherwise I would have tried to enroll in a university to learn about this stuff properly. I appreciate all your help! $\endgroup$
    – stats_noob
    Feb 21 at 3:58
  • 2
    $\begingroup$ All the same, when you have a question, you need to start your attempt to understand it with a high-quality resource about the topic, such as a textbook or a tutorial article. If you find a passage in the resource hard to understand, you'll need to make an attempt to familiarize yourself with the background material. If you're still stuck, use the search feature (here's a detailed explanation of how to get good results stats.meta.stackexchange.com/questions/5549/…). Once you've exhausted these steps, document your progress & ask a question. $\endgroup$
    – Sycorax ♦
    Feb 21 at 4:14

1 Answer 1

Reset to default
2
$\begingroup$

In the original paper by Hoerl and Kennard, in Technometrics in 1970, there is, if memory serves me well, a proof that, for $\lambda$ in some range, the ridge estimator outperforms the ordinary least squares estimator in terms of mean square error.

The problem, of course, is to ascertain what is the "right" $\lambda$ to use, but different heuristic methods have been proposed for that.

Improve this answer
$\endgroup$
1

Not the answer you're looking for? Browse other questions tagged or ask your own question.