9
$\begingroup$

I found that for a simple linear regression model, both OLS and maximum likelihood method (assuming Normal distribution) give the same output (parameter values). From this, can we say that OLS also make implicit assumption about the Normal distribution or vice-versa? I am not interested in why both produce same value but which one make less stringent assumption about the data?

$\endgroup$
5
$\begingroup$

OLS does not make the normality assumption for the model errors. OLS can be used under different distributional assumptions and the estimator will still make sense as the minimum variance unbiased estimator.

Maximum likelihood (ML) can also accommodate different distributions, but the distribution has to be chosen in advance. If the actual distribution appears to be not the same as the assumed distribution, ML estimator will no longer make sense as the estimator that maximizes the joint probability density of the model errors.

Thus we can say that in a particular application ML makes a more stringent assumption about the model errors than OLS does.

$\endgroup$
  • $\begingroup$ Thanks @RihchardHardy.....can you please explain what does it mean, "OLS can be used under different distributional assumptions" ? Because as per my knowledge OLS does not make any assumption about data, it only minimize sum of square of residuals. $\endgroup$ – Neeraj Dec 5 '15 at 10:00
  • $\begingroup$ You are right. That was intended to mean "Regardless of what distributional assumption, if any, you are making, OLS will be fine". $\endgroup$ – Richard Hardy Dec 5 '15 at 10:06
  • $\begingroup$ @thanks RichardHardy. It means that maximum likelihood is as good as assumption of distribution. But does it mean that for linear regression model OLS is always better than ML ? $\endgroup$ – Neeraj Dec 5 '15 at 10:49
  • 1
    $\begingroup$ Which is better depends on your objective function (or loss function). 1. If you want to maximize the joint probability density of errors, maximum likelihood is your choice (but you have to check how well the model residuals match the distributional assumption). Also, maximum likelihood may still work even if the error distribution is not matched; the keyword here is quasi maximum likelihood. 2. If you want to minimize the sum of squared residuals, go for OLS. $\endgroup$ – Richard Hardy Dec 5 '15 at 10:51
  • 1
    $\begingroup$ I think it is a question of interpretation. OLS and MLE have different goals. However, incidentally OLS and MLE are the same under normal distribution. Can this be interpreted as OLS unknowingly using the nomality assumption? Perhaps, but I would not put it that way. OLS is not based on it nor it has a goal of maximizing likelihood. I would call it a coincidence. In general, you may ask a new question rather than pose it in the comments. Comments should ideally be strongly related to the OP or the answer, while broader discussion may merit new questions and new answers. $\endgroup$ – Richard Hardy Dec 5 '15 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.