3
$\begingroup$

It seems that minimizing the sum of squared residuals (SSR) in linear regression is equivalent to minimizing MSE (both use true value - prediction) and OLS is the best estimator for minimizing SSR.

I also read that least squares can sometimes produce estimators with large variance under multicollinearity, in which case a biased estimator might produce a better MSE.

I am a bit confused why OLS is the best for SSR but sometimes is not the best on MSE, as these 2 metrics seemingly are proportional to each other.

Thanks.

$\endgroup$
0

2 Answers 2

1
$\begingroup$

You solve for the OLS coefficients by finding the coefficients that minimize the in-sample square loss (expressed as SSR or (R)MSE), so in-sample, nothing beats OLS.

When people say that biased estimators like the ridge regression estimator can produce better MSE, they mean on out-of-sample data. Fitting well in-sample might translate to fitting well out-of-sample, but it does not have to.

For instance, you could just play connect-the-dots with a scatterplot. Such a model perfectly predicts the connected dots. However, it has fit to the noise and will highly dependent on coincidences in the data. By regression to the mean, there is a sense in which, on new data, high points are likely to be lower and low points and likely to be higher. Your connect-the-dots mod with zero in-sample MSE will have poor performance if that happens.

A biased estimator combats this by sacrificing some in-sample fit in exchange for the possibility (but not assurance) of getting a better out-of-sample fit.

(Somewhat unrelated, you are correct to notice that SSR, MSE, and RMSE are equivalent in a sense. In a model comparison, a model outperforming another mode on one will outperform the competing model on the other two.)

$\endgroup$
0
$\begingroup$

This is because OLS only gives the best linear unbiased estimator (BLUE) under the assumptions of the Gauss-Markov theorem. It is only the best among linear, unbiased estimators. It doesn't mean that there can't be biased estimators that perform better than OLS.

$\endgroup$
1
  • $\begingroup$ Keep in mind that, in-sample, nothing beats OLS, and that is a consequence of (multivariable) calculus, not Gauss-Markov. In fact, OLS is unbeatable in-sample, even if the Gauss-Markov assumptions are grossly violated. $\endgroup$
    – Dave
    Dec 10, 2021 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.