2
$\begingroup$

In linear regression the assumption is that:

Y = (X)T.A + B + E

where, A and B are the parameters and E is the error term or Noise.

Wherever I read, the model assumes that the noise is normally distributed, with a constant variance.

What if the E ~ N(0, c^2) assumption is broken?

What if the variance is not constant? Why does the variance have to be constant?

What if E follows another distribution?

Could you show the impact of breaking these assumptions in terms of mathematical analysis?

$\endgroup$
5
$\begingroup$

Your question is not trivial and there are entire books written on these assumptions. As @Matthew Gunn has pointed out here are a few posts which summarize key concepts.

I will attempt to answer one part of your questions with a particular example

What if the variance is not constant?

If for example the error is normally distributed but the variance is not constant, i.e. $\epsilon \sim N(0, \sigma^2 \mathbf{W}^{-1})$ where $\mathbf{W}^{-1}$ is a matrix of weights, then you have a case of Heteroskedastic regression

Could you show the impact of breaking these assumptions in terms of mathematical analysis?

Derivation of the closed form solution for Heteroskedastic regression

Consider the case of multiple linear regression where the variance of the error terms is not constant. i.e. $\epsilon \sim N(0, \sigma^2 \textbf{W}^{-1})$. Intuitively, we would want to regression model to give less weight to observations with a large variance, and more weight to observations with small variance, as these are closer to the expected value we are trying to model. We can solve this problem using MLE as follows:

$$ \mathbf{Y = X \beta + \epsilon}$$

Recall the density of multivariate Gaussian in $p$ dimensions $\sim N(\mathbf{X \beta,\sigma^2 W^{-1}})$

$$ PDF = \frac{1}{(\text{det}(W^{-1})2 \pi)^{p/2} } \exp \mathbf{\left( - (Y - X \beta )^T W (Y - X \beta) /2 \right) }$$

The log-likelihood as a function of $\beta$ assuming $W$ is known is:

$$ \text{Constants} - \frac{1}{2} \mathbf{\left( (Y - X \beta )^T W (Y - X \beta) \right)} $$

To minimize the log-likelihood to get the optimum, recall that $W$ is diagonal and that we assume that no values are 0 or infinity. So the square root of $W$ is the square root of each value on the diagonal. $W = \sqrt{W} \sqrt{W}$ So we can re-write the equation as:

\begin{aligned} \text{Min}_\beta & \mathbf{ (Y - X \beta )^T W (Y - X \beta) } \\ \text{Min}_\beta & \mathbf{ (\sqrt{W}Y - \sqrt{W}X \beta )^T (\sqrt{W}Y - \sqrt{W}X \beta) } \\ & \text{Let $\tilde Y = \sqrt{W}Y$ and $\tilde X = \sqrt{W}X$} \\ \text{Min}_\beta & \mathbf{ (\tilde Y - \tilde X \beta )^T (\tilde Y - \tilde X \beta) } \\ \beta & = \mathbf{ (\tilde X^T \tilde X )^{-1} \tilde X^T \tilde Y} \\ & = \mathbf{ ( (\sqrt{W} X)^T \sqrt{W} X )^{-1} (\sqrt{W} X)^T \sqrt{W} Y} \\ & = \mathbf{ ( X^T (\sqrt{W} \sqrt{W} X )^{-1} X^T \sqrt{W} \sqrt{W} Y} \\ & = \mathbf{ ( X^T W X )^{-1} X^T W Y} \end{aligned}

Which is similar to the usual solution, with an added weight matrix $W$

Why does the variance have to be constant?

If you assume that the variance is constant when it is not, i.e. you fit an Ordinary Least Square model to heterosckedastic data, then the following will happen:

  • Your estimated line of best fit will still be consistent, i.e. as the number of data points increases to infinity your line will converge to the true one
  • Your estimated line of best fit will not be efficient, i.e. the rate of this convergence will be slower than what you get with an alternative model.
  • Your estimated variance will be wrong, mathematically it may be both biased and inconsistent

This means that any inference you make using your estimated variance will also be wrong, for example estimating the statistical significance of your model or plotting the confidence intervals.

What to do in practice is more subtle, and it depends on the extent to which your variance is not constant, the size of your dataset etc..

$\endgroup$
  • 3
    $\begingroup$ (+1) A comment though, generalized least squares is problematic in actual empirical work because you rarely know the covariance matrix for $\epsilon$ (what you call $\sigma^2 W^{-1}$). Of course you can estimate the covariance matrix first (i.e. do FGLS), but this estimation step adds problems. FGLS may actually be less efficient than regular OLS! Eg. in economics, effect sizes are often small relative to noise, and sample sizes may not be large enough for the asymptotic efficiency results of FGLS to even apply. $\endgroup$ – Matthew Gunn Jun 14 '18 at 20:40
  • $\begingroup$ Good point - it is not the first time that I come across this "trade-off" between the strength of assumptions and the sample size. Does this trade-off have a name and a more rigorous treatment ? $\endgroup$ – Xavier Bourret Sicotte Jun 14 '18 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.