This question has just bothered me hard enough to sign up here, so hello everyone. I have simulated, out of curiosity, simple linear data ($y = b_0 + b_1 x + \text{standard normal error}$) to let ML estimation compete against beta estimation through minimizing mean absolute error. Turns out that as long as the error term matches the ML-Distribution everything is fine. Even if I use uniformly distributed errors with mean 0 the ML estimator still gets reasonable results.

However, when I started using a ChiSquare with df=1, the ML estimate was just horrible (even for large sample sizes >200) and nowhere near the true values whereas the MAE-minimizing estimates for beta are still very reasonable even for small samples.

Now I know that ML has a lot of nice properties in theory but on the other side we might often not now the distribution of the error for real data. This keeps me thinking about applying ML for real world data where I will mostly prefer good estimates over theoretical attributes.


Are there some good books or papers that go into this on a theoretical basis?

  • 3
    $\begingroup$ Looks like what you are actually using is ordinary least squares (OLS) rather than maximum likelihood (ML), because ML with the assumption of normal errors produces OLS, and you seem to be using the assumption of normal errors througout in your "ML" estimation. So you are comparing minimizing MSE (which is what OLS does) with minimizing MAE under different distributions. $\endgroup$ – Richard Hardy Aug 18 '17 at 17:28
  • 1
    $\begingroup$ You already have an intercept term in your model, but if you have Chi-Squared distributed errors with one degree of freedom, their mean is non-zero. I think this leads to an issue with identifiability $\endgroup$ – klumbard Aug 18 '17 at 17:35
  • $\begingroup$ @klumbard: it shouldn't lead to an issue of identifiability, but one of heavy bias. $\endgroup$ – Cliff AB Aug 18 '17 at 17:37

It's a little hard to follow your example, but I think you are treating your error term as Chi-squared with one degree of freedom, when the true error term was normal and then performing maximum likelihood estimation with this distribution.

Note that this an extremely terrible assumption: for one, Chi-squared variables are strictly positive. This is going to induce extreme biases on the estimated regression coefficients.

In general, this is a very difficult problem: we know that MLE's have nice asymptotic optimality properties, but we also know that the likelihood is undoubtably misspecified. The effects of misspecification are not always straightforward. Statistical theory can give us some theory that says examining group means and simple linear regression still has somewhat nice properties, even if the data isn't normally distributed. But other than that, I'm not sure we have much more than handy-wavy arguments, i.e. "well, empirically, this looks like a Weibull distribution, so making a Weibull assumption probably isn't such a bad idea".

It's also worth noting that your MAE estimator is exactly equivalent to a maximum likelihood estimate in which we assume our error is from a Laplace distribution.

  • $\begingroup$ The last sentence is a good point. See also my comment above. $\endgroup$ – Richard Hardy Aug 18 '17 at 17:29

Thanks for your answers so far and sorry for being a little confusing in my explanation. What I did was always using a Normal Distribution for the ML-estimation while using different distributions (Normal, Uniform and Chi²) for the error in the Linear Data.

What still bothers me is the fact that we need a somewhat correct distribution assumption in order for ML to give reasonable results. However, in the real world we often won't have any clue what distribution to use and thus will be at risk of getting results that are far from truth. Now minimizing prediction error does not make any distributional assumptions and thus - as far as my understanding of ML goes - will be better than a mis-specified ML estimator. The only solution that I see right now would be to use a more flexible/generalized distribution for ML but this again could overfit my whole model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.