What is a complete list of the usual assumptions for linear regression?

What are the usual assumptions for linear regression?

Do they include:

1. a linear relationship between the independent and dependent variable
2. independent errors
3. normal distribution of errors
4. homoscedasticity

Are there any others?

-
I found this page quite helpful, for a review of the assumptions and the ways of testing them. – George Dontas Oct 3 '11 at 8:16
You can find a rather complete list in William Berry's little book on "Understanding Regression Assumptions": books.google.com/books/about/… – user5644 Oct 3 '11 at 9:43
While respondents have listed some good resources, it is a difficult question to answer in this format, and (many) books have been devoted solely to this topic. There is no cook book, nor should there be given the potential variety of situations that linear regression could encompass. – Andy W Oct 3 '11 at 12:49
Technically, (ordinary) linear regression is a model of the form $\mathbb{E}[Y_i] = \mathbf{X}_i \beta$, $Y_i$ iid. That simple mathematical statement encompasses all the assumptions. This leads me to think, @Andy W, that you may be interpreting the question more broadly, perhaps in the sense of the art and practice of regression. Your further thoughts about this might be useful here. – whuber Oct 3 '11 at 14:41
@mp You're right; I should have more carefully stated that $Y_i - \mathbf{X}_i\beta$ are iid. – whuber Oct 4 '11 at 13:45

The answer depends heavily on how do you define complete and usual. Suppose we write linear regression model in the following way:

$$y_i=\mathbf{x}_i'\beta+u_i$$

where $\mathbf{x}_i$ is the vector of predictor variables, $\beta$ is the parameter of interest, $y_i$ is the response variable, and $u_i$ are the disturbance. One of the possible estimates of $\beta$ is the least squares estimate:

$$\hat\beta=\textrm{argmin}_{\beta}\sum(y_i-\mathbf{x}_i\beta)^2=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum \mathbf{x}_iy_i$$

Now practically all of the textbooks deal with the assumptions when this estimate $\hat\beta$ has desirable properties, such as unbiasedness, consistency, efficiency, some distributional properties, etc.

Each of these properties requires certain assumptions, which are not the same. So the better question would be to ask which assumptions are needed for wanted properties of the LS estimate.

The properties I mention above require some probability model for regression. And here we have the situation where different models are used in different applied fields.

The simple case is to treat $y_i$ as an independent random variables, with $\mathbf{x}_i$ being non-random. I do not like the word usual, but we can say that this is the usual case in most applied fields (as far as I know).

Here is the list of some of the desirable properties of statistical estimates:

1. The estimate exists
2. Unbiasedness: $E\hat\beta=\beta$
3. Consistency: $\hat\beta\to \beta$ as $n\to\infty$ ($n$ here is the size of a data sample)
4. Efficiency: $Var(\hat\beta)$ is smaller than $Var(\tilde\beta)$ for alternative estimates $\tilde\beta$ of $\beta$
5. The ability to either approximate or calculate the distribution function of $\hat\beta$

Existence

Existence property might seem weird, but it is very important. In the definition of $\hat\beta$ we invert the matrix

$$\sum \mathbf{x}_i\mathbf{x}_i'$$

It is not guaranteed that the inverse of this matrix exists for all possible variants of $\mathbf{x}_i$. So we immediately get our first assumption:

Matrix $$\sum \mathbf{x}_i\mathbf{x}_i'$$ should be of full rank, i.e. invertible.

Unbiasedness

We have

$$E\hat\beta=\left(\sum \mathbf{x}_i\mathbf{x}_i\right)^{-1}\left(\sum \mathbf{x}_iEy_i\right)=\beta,$$

if

$$Ey_i=\mathbf{x}_i\beta.$$

We may number it the second assumption, but we may have stated it outright, since this is one of the natural ways to define linear relationship.

Note that to get unbiasedness we only require that $Ey_i=\mathbf{x}_i\beta$ for all $i$, and $\mathbf{x_i}$ are constants. Independence property is not required.

Consistency

For getting the assumptions for consistency we need to state more clearly what do we mean by $\to$. For sequences of random variables we have different modes of convergence: in probability, almost surely, in distribution and p-th moment sense. Suppose we want to get the convergence in probability. We can use either law of large numbers or directly the Chebyshev inequality (we employ the fact that $E\hat\beta=\beta$):

$$P(|\hat\beta-\beta|>\varepsilon)\le \frac{Var(\beta)}{\varepsilon^2}$$

Since convergence in probability means that the left hand term must vanish for any $\varepsilon>0$ as $n\to\infty$, we need that $Var(\beta)\to 0$ as $n\to\infty$. This is perfectly reasonable since with more data the precision with which we estimate $\beta$ should increase.

We have that

$$Var(\hat\beta)=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_i\sum_j\mathbf{x}_i\mathbf{x}_j'cov(y_i,y_j)\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}$$

Independence ensures that $cov(y_i,y_j)=0$, hence the expression simplifies to

$$Var(\hat\beta)=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_i\mathbf{x}_i\mathbf{x}_i'var(y_i)\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}$$

Now assume $Var(y_i)=const$, then

$$Var(\hat\beta)=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}Var(y_i)$$

Now if we additionaly require that $\frac{1}{n}\sum \mathbf{x}_i\mathbf{x}_i'$ is bounded for each $n$ we immediately get

$$Var(\beta)\to 0, n\to\infty.$$

So to get the consistency we assumed that there is no autocorrelation ($cov(y_i,y_j)=0$), t the variance $Var(y_i)$ is constant and $\mathbf{x}_i$ do not grow too much. The first assumption is satisfied if $y_i$ comes from independent sample.

Efficiency

The classic result is the Gauss-Markov theorem. The conditions for it is exactly the first two conditions for consistency and the condition for unbiasedness.

Distributional properties

If $y_i$ are normal we immediately get that $\hat\beta$ is normal since it is a linear combination of normal random variables. If we assume previous assumptions of independence, uncorrelatedness and constant variance we get that

$$\hat\beta\sim N\left(\beta,\sigma^2\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\right)$$

where $Var(y_i)=\sigma^2$.

If $y_i$ are not normal, but independent we can get approximate distribution of $\hat\beta$ thanks to central limit theorem. For this we need to assume that

$$\lim_n \frac{1}{n}\sum\mathbf{x}_i\mathbf{x}_i'\to A$$

for some matrix $A$. The constant variance for assymptotic normality is not required if we assume that

$$\lim_n\frac{1}{n}\sum\mathbf{x}_i\mathbf{x}_i'Var(y_i)\to B$$

Note that with constant variance of $y$ we cant that $B=\sigma^2A$. The central limit theorem then gives us the following result:

$$\sqrt{n}(\hat\beta-\beta)\to N(0,A^{-1}BA^{-1}).$$

So from this we see that independence and constant variance for $y_i$ and certain assumptions for $\mathbf{x}_i$ gives us a lot of useful properties for LS estimate $\hat\beta$.

The thing is that these assumptions can be relaxed. For example we required that $\mathbf{x}_i$ are not random variables. This assumption is not feasible in econometric applications. If we let $\mathbf{x}_i$ be random, we can get similar results if use conditional expectations and take into account the randomness of $\mathbf{x}_i$. The independence assumption also can be relaxed. We already demonstrated that sometimes only uncorrelatedness is needed. Even this can be further relaxed and it is still possible to show that the LS estimate will be consistent and assymptoticaly normal. See for example White's book for more details.

-
This is a really clear, excellent answer!! – Patrick S. Forscher Dec 5 '12 at 20:33
great answer..... – hypfco May 30 '15 at 19:58

There are a number of good answers here. It occurs to me that there is one assumption that has not been stated however (at least not explicitly). Specifically, a regression model assumes that $\mathbf X$ (the values of your explanatory / predictor variables) is fixed and known, and that all of the uncertainty in the situation exists within the $Y$ variable. In addition, this uncertainty is assumed to be sampling error only.

Here are two ways to think about this: If you are building an explanatory model (modeling experimental results), you know exactly what the levels of the independent variables are, because you manipulated / administered them. Moreover, you decided what those levels would be before you ever started gathering data. So you are conceptualizing all of the uncertainty in the relationship as existing within the response. On the other hand, if you are building a predictive model, it is true that the situation differs, but you still treat the predictors as though they were fixed and known, because, in the future, when you use the model to make a prediction about the likely value of $y$, you will have a vector, $\mathbf x$, and the model is designed to treat those values as though they are correct. That is, you will be conceiving of the uncertainty as being the unknown value of $y$.

These assumptions can be seen in the equation for a prototypical regression model: $$y_i = \beta_0 + \beta_1x_i + \varepsilon_i$$ A model with uncertainty (perhaps due to measurement error) in $x$ as well might have the same data generating process, but the model that's estimated would look like this: $$y_i = \hat\beta_0 + \hat\beta_1(x_i + \eta_i) + \hat\varepsilon_i,$$ where $\eta$ represents random measurement error. (Situations like the latter have led to work on errors in variables models; a basic result is that if there is measurement error in $x$, the naive $\hat\beta_1$ would be attenuated--closer to 0 than its true value, and that if there is measurement error in $y$, statistical tests of the $\hat\beta$'s would be underpowered, but otherwise unbiased.)

One practical consequence of the asymmetry intrinsic in the typical assumption is that regressing $y$ on $x$ is different from regressing $x$ on $y$. (See my answer here: What is the difference between doing linear regression on y with x versus x with y? for a more detailed discussion of this fact.)

-
What does it mean "fixed" | "random" in plain language? And how to distinguish between fixed and random effects(=factors)? I think that in my design there is 1 fixed known factor with 5 levels. Right? – stan Dec 9 '12 at 10:22
@stan, I recognize your confusion. Terminology in stats is often confusing & unhelpful. In this case, "fixed" is not quite the same as the fixed in 'fixed effects & random effects' (although they are related). Here, we're not talking about effects--we're talking about the $X$ data, ie your predictor / explanatory variables. The easiest way to understand the idea of your $X$ data being fixed is to think of a planned experiment. Before you have done anything, when you're designing the experiment, you decide what the levels of your explanatory will be, you don't discover them along the way. – gung Dec 9 '12 at 16:00
W/ predictive modeling, that's not quite true, but we will treat our $X$ data that way in the future, when we use the model to make predictions. – gung Dec 9 '12 at 16:00
Why do the βs and the ε have a hat in the bottom equation, but not in the top one? – user1205901 Apr 14 '15 at 8:41
@user1205901, the top model is of the data generating process, the bottom is your estimate of it. – gung Apr 14 '15 at 14:13

A comprehensive list of the assumption of the classical linear regressionmodel can be found here:

http://economictheoryblog.com/2015/04/01/ols_assumptions/

It is important to notice that the consequences in terms of validity of your model depend on which assumption is violated.

-

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Can you add a summary of the information contained in the link? The problem with link-only answers is that they can suffer from "link rot" - we are trying to build a long-term resource, and there's no guarantee the link will work in a few years time. Otherwise links are best suited to comments rather than answers. – Silverfish Oct 4 '15 at 11:49
Nonetheless, it is a good link, while note gone. – Marcelo Ventura May 8 at 23:04

As an aside, the four assumptions the OP mentions can easily be remembered using the acronym LINE:

• Linearity
• Independence
• Normality
• Equal Variance
-

It's all about what you wanna do with your model. Imagine if your errors were all positively skewed/non-normal. If you wanted to make a prediction interval, you could do better than using the t-distribution.. If your variance is smaller at smaller predicted values, again, you'd be making a prediction interval that's too big..

It's better to understand why the assumptions are there.

-

The following are the assumptions of Linear Regression analysis.

Correct specification. The linear functional form is correctly specified.

Strict exogeneity. The errors in the regression should have conditional mean zero.

No multicollinearity. The regressors in X must all be linearly independent.

Homoscedasticity which means that the error term has the same variance in each observation.

No autocorrelation: the errors are uncorrelated between observations.

Normality. It is sometimes additionally assumed that the errors have normal distribution conditional on the regressors.

I.i.d observations: $(x_i, y_i)$ is independent from, and has the same distribution as, $(x_j, y_j)$ for all $i\neq j$.