-1
$\begingroup$

In maximum likleihood, we believe that the y-variable is conditionally normally distributed. So this means that errors are also normally distributed.

In ols regression, things seem to be more algebra/geometry driven. I am trying to fit a line of best fit between some points. I have done this in high school in my vector algebra class ... and there was never any mention of normal distribution in fitting a line of best fit.

So how come the errors in OLS are needed to be normally distributed?

$\endgroup$
11
  • 1
    $\begingroup$ They are not (If by OLS you mean optimising a line to minimize the sum of squared residuals) $\endgroup$
    – Firebug
    Commented Sep 26, 2023 at 18:40
  • 2
    $\begingroup$ Hi: They only need to be normally distributed if you want to do inference and hypothesis testing. You can minimize the sum of the squared errors without the errors being normally distributed and still get a line of best fit. $\endgroup$
    – mlofton
    Commented Sep 26, 2023 at 18:42
  • 1
    $\begingroup$ Notice that, in the MLE framework, you are maximizing a likelihood so you need a likelihood function. The assumption of normally distributed errors results in the likelihood function. So, the two frameworks give the same coefficient estimates but under different assumptions. $\endgroup$
    – mlofton
    Commented Sep 26, 2023 at 18:44
  • $\begingroup$ thx! just to confirm ... ols does not need normal distribution errors? $\endgroup$
    – stats_noob
    Commented Sep 26, 2023 at 19:17
  • $\begingroup$ can you explain why they need to be normal for hypothesis test and inferences? $\endgroup$
    – stats_noob
    Commented Sep 26, 2023 at 19:17

2 Answers 2

1
$\begingroup$

In OLS, the errors do not have to have a normal distribution or even any particular distribution at all. All OLS does is solve the correspondence:

$$ \hat\beta_{OLS}\in\\ \underset{\beta=\left( \beta_0,\beta_1,\dots,\beta_p \right)}{\arg\min}\left\{ \overset{n}{\underset{i=1}{\sum}}\left( y_i-\left( \beta_0+\beta_1x_{i1}+\dots\beta_px_{ip} \right)\right)^2 \right\} $$

(I say that it is a correspondence instead of an equation because there does not have to be a unique solution $(\arg\min)$, such as if two features add up to a third.)

However, when you assume $iid$ Gaussian errors, the maximum likelihood estimate and OLS solution are equal.

That is the link.

$\endgroup$
6
  • 1
    $\begingroup$ +1. Viewing OLS as a gaussian glm is a popular error, but it is very freeing to understand that OLS is much more general than that with very desirable properties. $\endgroup$ Commented Sep 26, 2023 at 18:59
  • $\begingroup$ ok! so errors dont have to be normally distributed in OLS regression? this means i dont need to do all sorts of tests to see if this is true? $\endgroup$
    – stats_noob
    Commented Sep 26, 2023 at 19:14
  • $\begingroup$ I am still trying to understand .. imagine i do want to check if the errors are normally distributed ... I get the errors from the regression model ... and then see if these errors are normal? $\endgroup$
    – stats_noob
    Commented Sep 26, 2023 at 19:16
  • $\begingroup$ @jwolof No need to check for normality. The other assumptions of OLS are more important than the distribution of the errors. $\endgroup$ Commented Sep 26, 2023 at 19:19
  • $\begingroup$ @jwolof the assumptions about the normal distribution are needed (more as a sufficient condition and not so much as a neccesary condition) for inference about the estimates like computation of p-values or confidence intervals. $\endgroup$ Commented Sep 26, 2023 at 19:28
1
$\begingroup$

As was mentioned by several commentators, ordinary least-squares does not require normal errors. Only when you want to say something about the sampling distribution of the OLS estimator in finite samples, normality needs to be assumed or else the estimates are not necessarily t-distributed.

To some extent, though, we're splitting hairs. The negative log-likelihood, which is being minimized to obtain $\hat{\beta}_{MLE}$, is algebraically identical to the least-squares objective function whose minimization leads to $\hat{\beta}_{OLS}$. If you further don't take the normal likelihood "too literal" but rather understand it as a second-order approximation of the "true" likelihood, you've pretty much eliminated all conceptual differences between OLS and (quasi)-MLE.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.