What is the need of assumptions in linear regression?

Question

In linear regression, we make the following assumptions

The mean of the response, $E(Y_i)$, at each set of values of the predictors, $(x_{1i}, x_{2i},…)$, is a Linear function of the predictors.

The errors, $ε_i$, are Independent.

The errors, $ε_i$, at each set of values of the predictors, $(x_{1i}, x_{2i},…)$, are Normally distributed.

The errors, $ε_i$, at each set of values of the predictors, $(x_{1i}, x_{2i},…)$, have Equal variances (denoted $σ2$).

One of the ways we can solve linear regression is through normal equations, which we can write as

$$\theta = (X^TX)^{-1}X^TY$$

From a mathematical standpoint, the above equation only needs $X^TX$ to be invertible. So, why do we need these assumptions? I asked a few colleagues and they mentioned that it is to get good results and normal equations are an algorithm to achieve that. But in that case, how do these assumptions help? How does upholding them help in getting a better model?

Answer 1

You are correct - you do not need to satisfy these assumptions to fit a least squares line to the points. You need these assumptions to interpret the results. For example, assuming there was no relationship between an input $X_1$ and $Y$, what is the probability of getting a coefficient $\beta_1$ at least as great as what we saw from the regression?

Answer 2

Try the image of Anscombe's quartet from Wikipedia to get an idea of some of the potential issues with interpreting linear regression when some of those assumptions are clearly false: most of the basic descriptive statistics are the same in all four (and the individual $x_i$ values are identical in all but the bottom right)

Answer 3

You don't need those assumptions to fit a linear model. However, your parameter estimates could be biased or not having the minimum variance. Violating the assumptions will make yourself more difficult in interpreting the regression results, for example, constructing a confidence interval.

Answer 4

Ok, the answers so far go like this: If we violate the assumptions then bad things can happen. I believe that the interesting direction is: When all assumptions that we need (actually a little different from the ones above) are met, why and how can we be sure that linear regression is the best model?

I think the answer to that question goes like this: If we make the assumptions as in the answer of this question then we can compute the conditional density $p(y_i|x_i)$. From this we can compute $E[Y_i|X_i=x_i]$ (the factorization of the conditional expectation at $x_i$) and see that it is indeed the linear regression function. Then we use this in order to see that this is the best function with respect to the true risk.

Answer 5

The two key assumptions are

1. Independence of observations
2. Mean is not related to the variance

See The discussion in Julian Faraway's book.

If these are both true, OLS is surprisingly resistant to breaches in the other assumptions you have listed.