I found that for a simple linear regression model, both OLS and maximum likelihood method (assuming Normal distribution) give the same output (parameter values). From this, can we say that OLS also make implicit assumption about the Normal distribution or vice-versa? I am not interested in why both produce same value but which one make less stringent assumption about the data?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
13
OLS does not make a normality assumption for the model errors. OLS can be used under different distributional assumptions and the estimator will still make sense as the minimum variance linear unbiased estimator.
Maximum likelihood (ML) can also accommodate different distributions, but the distribution has to be chosen in advance. If the actual distribution appears to be different from the assumed distribution, ML estimator will no longer make sense as the estimator that maximizes the joint probability density of the data.
Thus we can say that in a particular application ML makes a more stringent assumption about the model errors than OLS does.
answered Dec 5, 2015 at 8:47
-
$\begingroup$ Thanks @RihchardHardy.....can you please explain what does it mean, "OLS can be used under different distributional assumptions" ? Because as per my knowledge OLS does not make any assumption about data, it only minimize sum of square of residuals. $\endgroup$– NeerajDec 5, 2015 at 10:00
-
$\begingroup$ You are right. That was intended to mean "Regardless of what distributional assumption, if any, you are making, OLS will be fine". $\endgroup$ Dec 5, 2015 at 10:06
-
$\begingroup$ @thanks RichardHardy. It means that maximum likelihood is as good as assumption of distribution. But does it mean that for linear regression model OLS is always better than ML ? $\endgroup$– NeerajDec 5, 2015 at 10:49
-
1$\begingroup$ Which is better depends on your objective function (or loss function). 1. If you want to maximize the joint probability density of errors, maximum likelihood is your choice (but you have to check how well the model residuals match the distributional assumption). Also, maximum likelihood may still work even if the error distribution is not matched; the keyword here is quasi maximum likelihood. 2. If you want to minimize the sum of squared residuals, go for OLS. $\endgroup$ Dec 5, 2015 at 10:51
-
1$\begingroup$ I think it is a question of interpretation. OLS and MLE have different goals. However, incidentally OLS and MLE are the same under normal distribution. Can this be interpreted as OLS unknowingly using the nomality assumption? Perhaps, but I would not put it that way. OLS is not based on it nor it has a goal of maximizing likelihood. I would call it a coincidence. In general, you may ask a new question rather than pose it in the comments. Comments should ideally be strongly related to the OP or the answer, while broader discussion may merit new questions and new answers. $\endgroup$ Dec 5, 2015 at 11:09