3
$\begingroup$

For the linear model

$$y_i=\beta_0 +\sum_{k=1}^{n}\beta_k x_{ik} + \epsilon_i$$

the parameter estimates are the same for the maximum likelihood method and the least square method (minimizing $\sum_{i=1}^n\epsilon_i^2$) if the errors are assumed iid normal. So I am thinking that if we use least squares, we are perhaps implicitly assigning normality to the errors. Is this correct - can it be shown that the least square method does this (without appealing to the maximum likelihood solution)?

$\endgroup$
  • 1
    $\begingroup$ Nope. But the converse almost works: normality (and heteroskedasticity) of errors plus one of @Aerandal's principles does imply ordinary least squares. $\endgroup$ – conjugateprior Sep 5 '14 at 20:35
8
$\begingroup$

There are simple examples where the MLE minimizes the SSE but the distribution is not normal. For starters, notice that the sample mean is the least-squares estimator of the location of a univariate sample $x_1,x_2,\ldots,x_n$, as it is the $\theta$ that minimizes $\sum(x_i-\theta)^2$.

Now consider the exponential distribution, $f(x) = \frac{1}{\theta} e^{-x/\theta}$, $x>0$; or the Poisson distribution with pmf $p(x) = e^{-\theta}\theta^x/x!$, $x=0,1,2,\ldots$. In both cases, given random samples from these distributions, the MLE of $\theta$ is the sample mean. So these are examples where the distribution is non-normal but the MLE is the least-squares estimator.

$\endgroup$
6
$\begingroup$

When you are doing the OLS estimation, you are just using a criteria for your estimation ( that is "minimize the sum of the squared residuals"), and so a particular method. You do not need anything (not normally distributed residuals nor anything) to do it and doing it does not imply anything. Whether this method is the best (for some definition of the best) depends on the assumptions you make.

Traditionally, in econometrics, it said to be the best estimator among the other ones if some assumptions are satisfied.

  • Without assuming normality of the residuals (but with other assumptions), OLS is the BLUE (Best Linear Unbiased Estimator).
  • If the residuals are normally distributed, OLS then become BUE (Best Unbiased Estimator)

Check here for the precise assumptions

$\endgroup$
  • 2
    $\begingroup$ +1 ... but of course, if you come to do inference (intervals or testing), you might assume normality then. $\endgroup$ – Glen_b -Reinstate Monica Sep 5 '14 at 16:03
  • $\begingroup$ So is it coincidence that the parameter estimates for MLE with normal errors are the same as OLS? $\endgroup$ – Comp_Warrior Sep 5 '14 at 21:01
  • $\begingroup$ I am less sure of what I will be saying there but my guess would be that it is the same because you are estimating the same model using the same criterion (minimizing square residuals), so you obtain the same estimate. $\endgroup$ – Anthony Martin Sep 7 '14 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.