8
$\begingroup$

This question seems fundamental enough that I'm convinced it has been answered here somewhere, but I haven't found it.

I understand that if the dependent variable in a regression is normally distributed, maximum likelihood and ordinary least squares produce the same parameter estimates.

When the dependent variable is not normally distributed, the OLS parameter estimates are no longer equivalent to MLE but they are still Best (minimum variance) Linear Unbiased estimates (BLUE).

So, what are the properties of MLE that make it desirable beyond what OLS has to offer (being BLUE)?

In other words, what do I lose if I can't say my OLS estimates are maximum likelihood estimates?

To motivate this question a little: I'm wondering why I would want to choose a regression model other than OLS in the presence of a clearly non-normal dependent variable.

$\endgroup$
  • $\begingroup$ Depends what you want from the model. Econometricians commonly want average in-sample marginal effect estimates, and OLS provides these (so long as you aren't bothered by additive separability assumptions). But if you want a model that describes other features of the underlying phenomenon, OLS won't work as well. For example, you might be interested in prediction out of sample, or want better estimates of uncertainty. $\endgroup$ – generic_user Nov 7 '17 at 17:31
  • $\begingroup$ It is worth mentioning that OLS is more sensitive to outliers, because the objective function uses squared error (so the problem gets worse the more extreme the deviation). This is why 'robust regression' techniques like M-estimation use MLE instead of OLS. $\endgroup$ – HEITZ Nov 7 '17 at 18:29
  • $\begingroup$ If the error term is not normal then t tests and F tests of coefficients may unreliable. Extreme skewness and extreme outliers are a particular problem. As practical matter this makes correct model specification harder, contributing to possible bias in the coefficient estimates (from miss-specification), and poor out of sample performance. $\endgroup$ – david25272 Nov 9 '17 at 23:38
7
$\begingroup$

As you move sufficiently far away from normality, all linear estimators may be arbitrarily bad.

Knowing that you can get the best of a bad lot (i.e. the best linear unbiased estimate) isn't much consolation.

If you can specify a suitable distributional model (ay, there's the rub), maximizing the likelihood has both a direct intuitive appeal - in that it "maximizes the chance" of seeing the sample you did actually see (with a suitable refinement of what we mean by that for the continuous case) and a number of very neat properties that are both theoretically and practically useful (e.g. relationship to the Cramer-Rao lower bound, equivariance under transformation, relationship to likelihood ratio tests and so forth). This motivates M-estimation for example.

Even when you can't specify a model, it is possible to construct a model for which ML is robust to contamination by gross errors in the conditional distribution of the response -- where it retains pretty good efficiency at the Gaussian but avoids the potentially disastrous impact of arbitrarily large outliers.

[That's not the only consideration with regression, since there's also a need for robustness to the effect of influential outliers for example, but it's a good initial step]


As a demonstration of the problem with even the best linear estimator, consider this comparison of slope estimators for regression. In this case there are 100 observations in each sample, x is 0/1, the true slope is $\frac12$ and errors are standard Cauchy. The simulation takes 1000 sets of simulated data and computes the least squares estimate of slope ("LS") as well as a couple of nonlinear estimators that could be used in this situation (neither is fully efficient at the Cauchy but they're both reasonable) - one is an L1 estimator of the line ("L1") and the second computes a simple L-estimate of location at the two values of x and fits a line joining them ("LE").

boxplots comparing the performance of a least squares slope estimator with some robust-to-gross-error estimates of slope suitable for this situation

The top part of the diagram is a boxplot of those thousand slope estimates for each simulation. The lower part is the central one percent (roughly, it is marked with a faint orange-grey box in the top plot) of that image "blown up" so we can see more detail. As we see the least squares slopes range from -771 to 1224 and the lower and upper quartiles are -1.24 and 2.46. The error in the LS slope was over 10 more than 10% of the time. The two nonlinear estimators do much better -- they perform fairly similarly to each other, none of the 1000 slope estimates in either case are more than 0.84 from the true slope and the median absolute error in the slope is in the ballpark of 0.14 for each (vs 1.86 for the least squares estimator). The LS slope has a RMSE of 223 and 232 times that of the L1 and LE estimators in this case (that's not an especially meaningful quantity, however as the LS estimator doesn't have a finite variance when you have Cauchy errors).

There are dozens of other reasonable estimators that might have been used here; this was simply a quick calculation to illustrate that even the best/most efficient linear estimators may not be useful. An ML estimator of the slope would perform better (in the MSE sense) than the two robust estimators used here, but in practice you'd want something with some robustness to influential points.

$\endgroup$
  • $\begingroup$ Well said. That makes a lot of sense. I assume that linear estimators still work quite well (maybe even better than non-linear estimators) when the dependent variable is non-normal but still symmetric. Is my intuition correct here? $\endgroup$ – Great38 Nov 9 '17 at 20:48
  • 1
    $\begingroup$ No, symmetry is not sufficient to rescue linear estimation. Consider Cauchy errors, for example. There are any number of adequate estimators but they're all nonlinear in the intended sense. $\endgroup$ – Glen_b Nov 9 '17 at 21:03
  • $\begingroup$ I have done a little simulation to illustrate this problem (of potentially arbitrarily bad performance) applies to symmetric error distributions -- see my edit. That simulation is for a symmetric error distribution. You can see how disastrous least squares can be in that case. Indeed even a tiny fraction of contamination with something that may have gross errors is a problem for it. Being BLUE may sometimes be of little value at all. If you know something about how your errors behave, it may be a good idea to use that knowledge ... ctd $\endgroup$ – Glen_b Nov 9 '17 at 22:35
  • $\begingroup$ ctd... (via ML, say, even if you supplement it with a little robustness in case you're wrong; like bounding the influence function of the resulting M-estimator) and if you know nothing -- not enough to assume a likelihood at all -- it's not necessarily worth the potential risk in assuming that a linear estimator will be a great choice. I wouldn't say "never use least squares" (I use it fairly regularly yet have been keenly aware of how bad it may potentially perform) -- but BLUE on its own doesn't necessarily imply that it's a good choice. $\endgroup$ – Glen_b Nov 9 '17 at 22:57
1
$\begingroup$

In the case of normally distributed data, OLS converges with the MLE, a solution which is BLUE (in that point). Once out of normal, OLS isn't BLUE anymore (in the terms of Gauss-Markov theorem) - this is because OLS looks to minimize the SSR whereas GMT defines BLUE in terms of minimal SE. See more here.

Generally speaking, given that a MLE exists (google for 'MLE failure' or for cases where MLE doesn't exist), it is easier to adjust it, either for minimizing the variance or making it unbiased (and therefore comparable to other estimators).

$\endgroup$
  • 3
    $\begingroup$ The dependent variable doesn't need to be normal for OLS to be BLUE: en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem $\endgroup$ – Great38 Nov 7 '17 at 20:29
  • 1
    $\begingroup$ ... in addition, with normally distributed data, OLS = the MLE, it doesn't converge to it. Your second paragraph is also quite unclear... easier to adjust the MLE than what? $\endgroup$ – jbowman Nov 7 '17 at 20:51
  • $\begingroup$ OLS is still BLUE outside of normality; the problem is that BLUE (and in particular, the L) itself isn't necessarily a useful thing to have. $\endgroup$ – Glen_b May 21 '18 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.