When OLS overfits observed data, does it give skewed distribution of estimates?


The image below from an answer to another question here might help.

Why does regularization wreck orthogonality of predictions and residuals in linear regression?

least squares vs regularized

  • The OLS solution is an orthogonal projection of the observations into a subspace defined by the model.

  • The ridge regression solution is also a sort of orthogonal projection, but now the subspace is restricted by the regularisation conditions.

With OLS you get that this orthogonal projection is on a flat surface and is like splitting the random noise (which is spherically symmetric in tha case of white noise) into two orthogonal parts.

With ridge regression the projection is on a curved surface (the same is true for non-linear least squares) and indeed, this will make the sample distribution of the estimate slightly skewed. With larger sample size, this skewness will reduce because the curvature of the surface becomes less.

I have no illustration of this, but you can imagine it as somewhat similar to the Delta method approaching a normal distribution. Since ridge regression is a linear estimator (a weighted sum of the observations) I guess that there might be some matrix formulation of the estimate by which one can proof that ridge regression approaches a normal distribution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.