1
$\begingroup$

We say that under assumptions of the Gauss-Markov theorem, OLS is BLUE. The Gauss-Markov theorem doesn't mention the normality of errors.

  1. If the errors are distributed as per the Laplace distribution, then is OLS still BLUE as per Gauss Markov Theorem?

  2. Using MLE estimate for Laplacian errors, we would get that the optimal linear estimator would minimize the MAE, not the SSE. So MLE estimate is no longer the OLS estimate when errors are not normal. What does this mean? which is more appropriate to use?

$\endgroup$

1 Answer 1

0
$\begingroup$
  1. OLS is BLUE regardless of the distribution of the errors, as long as they have a finite variance. However, the "Best" in BLUE refers to a specific criterion: variance, or, equivalently in the case of unbiased estimators, mean squared error... but only guaranteed to be "Best" within the class of linear unbiased estimators.

  2. The maximum likelihood estimator for a Laplace distribution with unit scale is based on maximizing the following log-likelihood:

$$\max_{\theta} -\sum \vert x_i - \theta\vert$$

which is equivalent to minimizing the absolute error, not the squared error.

Which one is more appropriate depends on your problem's specifics. If you don't have a clear, externally driven loss function, the MLE is typically the preferred choice, as the MLE is asymptotically optimal in terms of MSE. Note also that the MLE is somewhat less sensitive to observations far from $\theta$ than OLS, which makes it somewhat more robust (in an informal sense of the term.) If you do have a loss function, though, minimizing it would be the better choice.

Edit in response to comments:

Let's compare the performance of OLS and the MLE in a straightforward case: no independent variables, just a constant term. In this case, the OLS is the sample mean, and the MLE is the sample median. We calculate the squared sample error for five sample sizes - $5, 10, 20, 50, 100$ - and find the ratio of the mean squared error over 10,000 observations:

library(extraDistr)

sample_sizes <- c(5,10,20,50,100)
mle_var <- ols_var <- rep(0,5)

for (i in seq_along(sample_sizes)) {
    for (j in 1:10000) {
        x <- rlaplace(sample_sizes[i])
        ols_var[i] <- ols_var[i] + mean(x)^2
        mle_var[i] <- mle_var[i] + median(x)^2 
    }
}

with result:

> ols_var / mle_var
[1] 1.140804 1.363045 1.517283 1.634981 1.713133

demonstrating that not only is the MLE a better estimator, in the mean squared error sense, for very small samples, it gets better still as the sample size increases.

$\endgroup$
2
  • $\begingroup$ Thanks @jbowman. Small question here: Isn't MLE asymptotically efficient? Efficiency means least variance of the estimator right? and so it seems to me that the MLE estimate should be "best" and not OLS. And therefore, we should find beta coefficients of regression by minimising MAE and not MSE. What part of my reasoning is incorrect? $\endgroup$ Commented 2 days ago
  • $\begingroup$ OLS is the best linear unbiased estimator; MLE need not be linear or unbiased, so it can be better - and, as you point out, is asymptotically efficient, which OLS is not unless it happens to coincide with the MLE (as in the Gaussian error case.) In this case, the MLE is not linear. Note that in small samples, the MLE can be worse than a lot of other estimators, but that does tend to change as $n \to \infty$. $\endgroup$
    – jbowman
    Commented 2 days ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.