13
$\begingroup$

This question is inspired from the long discussion in comments here: How does linear regression use the normal distribution?

In the usual linear regression model, for simplicity here written with only one predictor: $$ Y_i = \beta_0 + \beta_1 x_i + \epsilon_i $$ where the $x_i$ are known constants and $\epsilon_i$ are zero-mean independent error terms. If we in addition assume normal distributions for the errors, then the usual least squares estimators and the maximum likelihood estimators of $\beta_0, \beta_1$ are identical.

So my easy question: do there exist any other distribution for the error terms such that the mle are identical with the ordinary least squaeres estimator? The one implication is easy to show, the other one not so.

$\endgroup$
  • 1
    $\begingroup$ (+1) It would need to be a distribution centered around zero, and it would seem it would help if it were a symmetric one. Some candidates that come to mind, like the t- or the Laplace distribution do not seem to do the trick as the MLE is, even in the constant only case, not available in closed form or given by the median, respectively. $\endgroup$ – Christoph Hanck Sep 22 '15 at 13:56
  • $\begingroup$ see also stats.stackexchange.com/questions/99014/…, it seems there is only so much to find $\endgroup$ – Christoph Hanck Sep 22 '15 at 13:59
  • $\begingroup$ I'm sure the answer is no. May be hard to write an rigorous proof however. $\endgroup$ – Gordon Smyth Jul 26 '17 at 17:10
11
$\begingroup$

In maximum likelihood estimation, we calculate

$$\hat \beta_{ML}: \sum \frac {\partial \ln f(\epsilon_i)}{\partial \beta} = \mathbf 0 \implies \sum \frac {f'(\epsilon_i)}{f(\epsilon_i)}\mathbf x_i = \mathbf 0$$

the last relation taking into account the linearity structure of the regression equation.

In comparison , the OLS estimator satisfies

$$\sum \epsilon_i\mathbf x_i = \mathbf 0$$

In order to obtain identical algebraic expressions for the slope coefficients we need to have a density for the error term such that

$$\frac {f'(\epsilon_i)}{f(\epsilon_i)} = \pm \;c\epsilon_i \implies f'(\epsilon_i)= \pm \;c\epsilon_if(\epsilon_i)$$

These are differential equations of the form $y' = \pm\; xy$ that have solutions

$$\int \frac 1 {y}dy = \pm \int x dx\implies \ln y = \pm\;\frac 12 x^2$$

$$ \implies y = f(\epsilon) = \exp\left \{\pm\;\frac 12 c\epsilon^2\right\}$$

Any function that has this kernel and integrates to unity over an appropriate domain, will make the MLE and OLS for the slope coefficients identical. Namely we are looking for

$$g(x)= A\exp\left \{\pm\;\frac 12 cx^2\right\} : \int_a^b g(x)dx =1$$

Is there such a $g$ that is not the normal density (or the half-normal or the derivative of the error function)?

Certainly. But one more thing one has to consider is the following: if one uses the plus sign in the exponent, and a symmetric support around zero for example, one will get a density that has a unique minimum in the middle, and two local maxima at the boundaries of the support.

$\endgroup$
  • $\begingroup$ Great answer (+1), but if one uses a plus sign in the function, is it even a density? It would appear then that the function has infinite integral and thus cannot be normalised to a density function. If that is the case, we are left only with the normal distribution. $\endgroup$ – Reinstate Monica Mar 26 '18 at 5:46
  • 1
    $\begingroup$ @Ben Thanks. It seems that you are implicitly assume that the range of the random variable will be plus/minus infinity. But we can define an rv to range in a bounded interval, in which case we can very well use the plus sign. This is why in my expressions I used as limits of integration $(a,b)$. $\endgroup$ – Alecos Papadopoulos Mar 26 '18 at 7:46
  • $\begingroup$ That's true - I was assuming that. $\endgroup$ – Reinstate Monica Mar 26 '18 at 7:50
5
$\begingroup$

If we define the OLS as the solution to $$\arg_{\beta_0,\beta_1}\min\sum_{i=1}^n (y_i-\beta_0-\beta_1x_i)^2$$ any density $f(y|x,\beta_0,\beta_1)$ such that $$\arg_{\beta_0,\beta_1}\min\sum_{i=1}^n \log\{f(y_i|x_i,\beta_0,\beta_1)\}=\arg_{\beta_0,\beta_1}\min\sum_{i=1}^n (y_i-\beta_0-\beta_1x_i)^2$$is acceptable. This means for instance that densities of the form $$f(y|x,\beta_0,\beta_1)=f_0(y|x)\exp\{-\omega(y_i-\beta_0-\beta_1x_i)^2\}$$ are acceptable since the factor $f_0(y|x)$ does not depend on the parameter $(\beta_0,\beta_1)$. There is therefore an infinity of such distributions.

Another setting where both estimators coincide is when the data comes from a spherically symmetric distribution, namely when the (vector) data $\mathbf{y}$ has conditional density$$h(||\mathbf{y}-\mathbf{X}\beta||)$$ with $h(\cdot)$ a decreasing function. (In this case the OLS is still available although the assumption of the independence of the $\epsilon_i$'s only holds in the Normal case.)

$\endgroup$
  • 1
    $\begingroup$ This does not look correct to me. If you use a different spherically symmetric distribution, wouldn't that lead to minimisation of a different function of the norm than the square (thus not being least-squares estimation)? $\endgroup$ – Reinstate Monica Mar 26 '18 at 5:49
1
$\begingroup$

I didn't know about this question until @Xi'an just updated with an answer. There is a more generic solution. Exponential family distributions with some parameters fixed yield to Bregman divergences. For such distributions mean is the minimizer. OLS minimizer is also the mean. Therefore for all such distributions they should coincide when the linear functional is linked to the mean parameter.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.75.6958&rep=rep1&type=pdf

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.