Ordinary Least Squares regression is defined as minimizing the sum of squared errors. So after doing this regression (OLS) then what is the purpose of optimizing SSE (or MSE, RMSE etc.) if linear regression already revolves around optimizing the position of the best fit line that minimizes the sum of squared errors? To further clarify when I say optimize SSE I am referring to using gradient descent ( or other algorithms) to optimize it.
7
-
$\begingroup$ So you know how good “best” is. $\endgroup$– Arya McCarthyCommented May 27, 2021 at 13:57
-
$\begingroup$ What do you mean by optimizing? Do you mean something like model selection or tuning hyperparameters? // You're right that the usual OLS fit $\hat{\beta} = (X^TX)^{-1}X^Ty$ gives the minimum SSE (MSE, RMSE) for the given $X$ and $y$. $\endgroup$– DaveCommented May 27, 2021 at 14:06
-
$\begingroup$ In what context are you asking? E.g. the OLS fit does so for training data, but perhaps someone wants to evaluate performance on a new dataset? Or perhaps they want to know what RMSE is (not just that it's minimized)? $\endgroup$– BjörnCommented May 27, 2021 at 14:16
-
$\begingroup$ To clarify what I mean is, I have witnessed people using gradient descent (or other optimization algorithms) to minimize SSE, but I am confused because I thought one has already minimized the sum of squared errors when using OLS. I hope that clarifies the question. $\endgroup$– yonasbosonCommented May 27, 2021 at 14:17
-
1$\begingroup$ That's just a demonstration of gradient descent in a straightforward setting; the reason for doing it is to teach gradient descent. Its power is in difficult situations like neural networks where the parameter estimates don't have a closed-form solution like $\hat{\beta}=(X^TX)^{-1}X^Ty$. // I would have liked to have seen the author compare the gradient descent parameter estimates to those from $\hat{\beta}=(X^TX)^{-1}X^Ty$ to confirm that they are the same (it better be). $\endgroup$– DaveCommented May 27, 2021 at 14:38
|
Show 2 more comments
1 Answer
$\begingroup$
$\endgroup$
When you're doing regression (of any type), even though you are minimizing the sum of the squared errors, that only means that your result is probably close to correct. And here, 'probably' and 'close' are fuzzy and vague. Extra measures like MSD/MSE, SSE, RMSE, RX2S, and so on are there to help you determine how much you want to trust the regression.
You're almost never interested in an actual observed value when doing regression -- you typically either want to interpolate or extrapolate some unobserved value based on a limited sample, or discover trends in your data (which, themselves, contain error and uncertainty). Even you somehow managed to measure and record the exact values for a variable without any error, and those points perfectly fit a line (or hyperbola, or whatever function you choose), you want some way of measuring that. These kinds of metrics that you're asking about give you a lot of information about the model you've constructed to describe your data set.
Is the model appropriate for your data? Can you trust your results? Are the residuals huge and not distributed randomly (meaning linear regression isn't giving you a good picture of the true behavior of your system / relationship[s] between your variables)? Are your findings statistically significant? Does your model describe the data better than some other model you (or someone else) has
