In linear regression, we make the following assumptions

  • The mean of the response, $E(Y_i)$, at each set of values of the predictors, $(x_{1i}, x_{2i},…)$, is a Linear function of the predictors.
  • The errors, $ε_i$, are Independent.
  • The errors, $ε_i$, at each set of values of the predictors, $(x_{1i}, x_{2i},…)$, are Normally distributed.
  • The errors, $ε_i$, at each set of values of the predictors, $(x_{1i}, x_{2i},…)$, have Equal variances (denoted $σ2$).
  • One of the ways we can solve linear regression is through normal equations, which we can write as

    $$\theta = (X^TX)^{-1}X^TY$$

    From a mathematical standpoint, the above equation only needs $X^TX$ to be invertible. So, why do we need these assumptions? I asked a few colleagues and they mentioned that it is to get good results and normal equations are an algorithm to achieve that. But in that case, how do these assumptions help? How does upholding them help in getting a better model?

    • 2
      Normal distribution is needed to calculate coefficient confidence intervals using usual formulas. Other formulas of CI calculation (I think it was White) allow non-normal distribution. – keiv.fly Aug 15 at 8:30
    • You do not always need those assumptions for the model to work. In neural networks you have linear regressions inside and they minimize rmse just like the formula you provided, but most likely none of the assumptions hold. No normal distribution, no equal variance, no linear function, even the errors can be dependent. – keiv.fly Aug 15 at 8:43
    • 2
    • 1
      @Alexis The independent variables being i.i.d. definitely isn't an assumption (and the dependent variable being iid also isn't an assumption - imagine if we assumed that the response was iid then it would be pointless to do anything beyond estimating the mean). And the "no omitted variables" isn't really an additional assumption although it's good to avoid omitting variables - the first assumption listed is really what takes care of that. – Dason Aug 15 at 14:57
    • 1
      @Dason I think my link provides a pretty strong example of "no omitted variables" being requisite for valid interpretation. I also think i.i.d. (conditional on the predictors, yes) is necessary, with random walks providing an excellent example of where non-i.i.d. estimation can fail (ever resorting to estimating only the mean). – Alexis Aug 15 at 17:02
    up vote 19 down vote accepted

    You are correct - you do not need to satisfy these assumptions to fit a least squares line to the points. You need these assumptions to interpret the results. For example, assuming there was no relationship between an input $X_1$ and $Y$, what is the probability of getting a coefficient $\beta_1$ at least as great as what we saw from the regression?

    Try the image of Anscombe's quartet from Wikipedia to get an idea of some of the potential issues with interpreting linear regression when some of those assumptions are clearly false: most of the basic descriptive statistics are the same in all four (and the individual $x_i$ values are identical in all but the bottom right)

    https://upload.wikimedia.org/wikipedia/commons/thumb/e/ec/Anscombe%27s_quartet_3.svg/1280px-Anscombe%27s_quartet_3.svg.png

    You don't need those assumptions to fit a linear model. However, your parameter estimates could be biased or not having the minimum variance. Violating the assumptions will make yourself more difficult in interpreting the regression results, for example, constructing a confidence interval.

    Ok, the answers so far go like this: If we violate the assumptions then bad things can happen. I believe that the interesting direction is: When all assumptions that we need (actually a little different from the ones above) are met, why and how can we be sure that linear regression is the best model?

    I think the answer to that question goes like this: If we make the assumptions as in the answer of this question then we can compute the conditional density $p(y_i|x_i)$. From this we can compute $E[Y_i|X_i=x_i]$ (the factorization of the conditional expectation at $x_i$) and see that it is indeed the linear regression function. Then we use this in order to see that this is the best function with respect to the true risk.

    The two key assumptions are

    1. Independence of observations
    2. Mean is not related to the variance

    See The discussion in Julian Faraway's book.

    If these are both true, OLS is surprisingly resistant to breaches in the other assumptions you have listed.

    Your Answer

     
    discard

    By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

    Not the answer you're looking for? Browse other questions tagged or ask your own question.