Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

What are the usual assumptions for linear regression?

Do they include:

  1. a linear relationship between the independent and dependent variable
  2. independent errors
  3. normal distribution of errors
  4. homoscedasticity

Are there any others?

share|improve this question
    
I found this page quite helpful, for a review of the assumptions and the ways of testing them. –  George Dontas Oct 3 '11 at 8:16
1  
You can find a rather complete list in William Berry's little book on "Understanding Regression Assumptions": books.google.com/books/about/… –  user5644 Oct 3 '11 at 9:43
2  
While respondents have listed some good resources, it is a difficult question to answer in this format, and (many) books have been devoted solely to this topic. There is no cook book, nor should there be given the potential variety of situations that linear regression could encompass. –  Andy W Oct 3 '11 at 12:49
    
Technically, (ordinary) linear regression is a model of the form $\mathbb{E}[Y_i] = \mathbf{X}_i \beta$, $Y_i$ iid. That simple mathematical statement encompasses all the assumptions. This leads me to think, @Andy W, that you may be interpreting the question more broadly, perhaps in the sense of the art and practice of regression. Your further thoughts about this might be useful here. –  whuber Oct 3 '11 at 14:41
    
I assumed (perhaps wrongly) that the assumptions the OP talks about are in regards to making valid inferences based on the $\beta$ estimates, which require greater constraints than those that simply allow the $\beta$ to be identifiable (as mentioned by @whuber). It would require clarification from tony though as to whether my assumption is correct (and if it is my first comment still stands, in that it is so broad it would be difficult (but not impossible) to write an answer with a scope that wide). –  Andy W Oct 3 '11 at 17:47
show 3 more comments

6 Answers 6

The answer depends heavily on how do you define complete and usual. Suppose we write linear regression model in the following way:

$$y_i=\mathbf{x}_i'\beta+u_i$$

where $\mathbf{x}_i$ is the vector of predictor variables, $\beta$ is the parameter of interest, $y_i$ is the response variable, and $u_i$ are the disturbance. One of the possible estimates of $\beta$ is the least squares estimate:

$$\hat\beta=\textrm{argmin}_{\beta}\sum(y_i-\mathbf{x}_i\beta)^2=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum \mathbf{x}_iy_i$$

Now practically all of the textbooks deal with the assumptions when this estimate $\hat\beta$ has desirable properties, such as unbiasedness, consistency, efficiency, some distributional properties, etc.

Each of these properties requires certain assumptions, which are not the same. So the better question would be to ask which assumptions are needed for wanted properties of the LS estimate.

The properties I mention above require some probability model for regression. And here we have the situation where different models are used in different applied fields.

The simple case is to treat $y_i$ as an independent random variables, with $\mathbf{x}_i$ being non-random. I do not like the word usual, but we can say that this is the usual case in most applied fields (as far as I know).

Here is the list of some of the desirable properties of statistical estimates:

  1. The estimate exists
  2. Unbiasedness: $E\hat\beta=\beta$
  3. Consistency: $\hat\beta\to \beta$ as $n\to\infty$ ($n$ here is the size of a data sample)
  4. Efficiency: $Var(\hat\beta)$ is smaller than $Var(\tilde\beta)$ for alternative estimates $\tilde\beta$ of $\beta$
  5. The ability to either approximate or calculate the distribution function of $\hat\beta$

Existence

Existence property might seem weird, but it is very important. In the definition of $\hat\beta$ we invert the matrix

$$\sum \mathbf{x}_i\mathbf{x}_i'$$

It is not guaranteed that the inverse of this matrix exists for all possible variants of $\mathbf{x}_i$. So we immediately get our first assumption:

Matrix $$\sum \mathbf{x}_i\mathbf{x}_i'$$ should be of full rank, i.e. invertible.

Unbiasedness

We have

$$E\hat\beta=\left(\sum \mathbf{x}_i\mathbf{x}_i\right)^{-1}\left(\sum \mathbf{x}_iEy_i\right)=\beta,$$

if

$$Ey_i=\mathbf{x}_i\beta.$$

We may number it the second assumption, but we may have stated it outright, since this is one of the natural ways to define linear relationship.

Note that to get unbiasedness we only require that $Ey_i=\mathbf{x}_i\beta$ for all $i$, and $\mathbf{x_i}$ are constants. Independence property is not required.

Consistency

For getting the assumptions for consistency we need to state more clearly what do we mean by $\to$. For sequences of random variables we have different modes of convergence: in probability, almost surely, in distribution and p-th moment sense. Suppose we want to get the convergence in probability. We can use either law of large numbers or directly the Chebyshev inequality (we employ the fact that $E\hat\beta=\beta$):

$$P(|\hat\beta-\beta|>\varepsilon)\le \frac{Var(\beta)}{\varepsilon^2}$$

Since convergence in probability means that the left hand term must vanish for any $\varepsilon>0$ as $n\to\infty$, we need that $Var(\beta)\to 0$ as $n\to\infty$. This is perfectly reasonable since with more data the precision with which we estimate $\beta$ should increase.

We have that

$$Var(\hat\beta)=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_i\sum_j\mathbf{x}_i\mathbf{x}_j'cov(y_i,y_j)\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}$$

Independence ensures that $cov(y_i,y_j)=0$, hence the expression simplifies to

$$Var(\hat\beta)=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_i\mathbf{x}_i\mathbf{x}_i'var(y_i)\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}$$

Now assume $Var(y_i)=const$, then

$$Var(\hat\beta)=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}Var(y_i)$$

Now if we additionaly require that $\frac{1}{n}\sum \mathbf{x}_i\mathbf{x}_i'$ is bounded for each $n$ we immediately get

$$Var(\beta)\to 0, n\to\infty.$$

So to get the consistency we assumed that there is no autocorrelation ($cov(y_i,y_j)=0$), t the variance $Var(y_i)$ is constant and $\mathbf{x}_i$ do not grow too much. The first assumption is satisfied if $y_i$ comes from independent sample.

Efficiency

The classic result is the Gauss-Markov theorem. The conditions for it is exactly the first two conditions for consistency and the condition for unbiasedness.

Distributional properties

If $y_i$ are normal we immediately get that $\hat\beta$ is normal since it is a linear combination of normal random variables. If we assume previous assumptions of independence, uncorrelatedness and constant variance we get that

$$\hat\beta\sim N\left(\beta,\sigma^2\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\right)$$

where $Var(y_i)=\sigma^2$.

If $y_i$ are not normal, but independent we can get approximate distribution of $\hat\beta$ thanks to central limit theorem. For this we need to assume that

$$\lim_n \frac{1}{n}\sum\mathbf{x}_i\mathbf{x}_i'\to A$$

for some matrix $A$. The constant variance for assymptotic normality is not required if we assume that

$$\lim_n\frac{1}{n}\sum\mathbf{x}_i\mathbf{x}_i'Var(y_i)\to B$$

Note that with constant variance of $y$ we cant that $B=\sigma^2A$. The central limit theorem then gives us the following result:

$$\sqrt{n}(\hat\beta-\beta)\to N(0,A^{-1}BA^{-1}).$$

So from this we see that independence and constant variance for $y_i$ and certain assumptions for $\mathbf{x}_i$ gives us a lot of useful properties for LS estimate $\hat\beta$.

The thing is that these assumptions can be relaxed. For example we required that $\mathbf{x}_i$ are not random variables. This assumption is not feasible in econometric applications. If we let $\mathbf{x}_i$ be random, we can get similar results if use conditional expectations and take into account the randomness of $\mathbf{x}_i$. The independence assumption also can be relaxed. We already demonstrated that sometimes only uncorrelatedness is needed. Even this can be further relaxed and it is still possible to show that the LS estimate will be consistent and assymptoticaly normal. See for example White's book for more details.

share|improve this answer
    
This is a really clear, excellent answer!! –  Patrick S. Forscher Dec 5 '12 at 20:33
add comment

There are a number of good answers here. It occurs to me that there is one assumption that has not been stated however (at least not explicitly). Specifically, a regression model assumes that $\mathbf X$ (the values of your explanatory / predictor variables) is fixed and known, and that all of the uncertainty in the situation exists within the $Y$ variable. In addition, this uncertainty is assumed to be sampling error only.

Here are two ways to think about this: If you are building an explanatory model (modeling experimental results), you know exactly what the levels of the independent variables are, because you manipulated / administered them. Moreover, you decided what those levels would be before you ever started gathering data. So you are conceptualizing all of the uncertainty in the relationship as existing within the response. On the other hand, if you are building a predictive model, it is true that the situation differs, but you still treat the predictors as though they were fixed and known, because, in the future, when you use the model to make a prediction about the likely value of $y$, you will have a vector, $\mathbf x$, and the model is designed to treat those values as though they are correct. That is, you will be conceiving of the uncertainty as being the unknown value of $y$.

These assumptions can be seen in the equation for a prototypical regression model: $$ y_i = \beta_0 + \beta_1x_i + \varepsilon_i $$ A model with uncertainty (perhaps due to measurement error) in $x$ as well might have the same data generating process, but the model that's estimated would look like this: $$ y_i = \hat\beta_0 + \hat\beta_1(x_i + \eta_i) + \hat\varepsilon_i, $$ where $\eta$ represents random measurement error. (Situations like the latter have led to work on errors in variables models; a basic result is that if there is measurement error in $x$, the naive $\hat\beta_1$ would be attenuated--closer to 0 than its true value, and that if there is measurement error in $y$, statistical tests of the $\hat\beta$'s would be underpowered, but otherwise unbiased.)

One practical consequence of the asymmetry intrinsic in the typical assumption is that regressing $y$ on $x$ is different from regressing $x$ on $y$. (See my answer here: What is the difference between doing linear regression on y with x versus x with y? for a more detailed discussion of this fact.)

share|improve this answer
    
What does it mean "fixed" | "random" in plain language? And how to distinguish between fixed and random effects(=factors)? I think that in my design there is 1 fixed known factor with 5 levels. Right? –  stan Dec 9 '12 at 10:22
    
@stan, I recognize your confusion. Terminology in stats is often confusing & unhelpful. In this case, "fixed" is not quite the same as the fixed in 'fixed effects & random effects' (although they are related). Here, we're not talking about effects--we're talking about the $X$ data, ie your predictor / explanatory variables. The easiest way to understand the idea of your $X$ data being fixed is to think of a planned experiment. Before you have done anything, when you're designing the experiment, you decide what the levels of your explanatory will be, you don't discover them along the way. –  gung Dec 9 '12 at 16:00
    
W/ predictive modeling, that's not quite true, but we will treat our $X$ data that way in the future, when we use the model to make predictions. –  gung Dec 9 '12 at 16:00
add comment

It's all about what you wanna do with your model. Imagine if your errors were all positively skewed/non-normal. If you wanted to make a prediction interval, you could do better than using the t-distribution.. If your variance is smaller at smaller predicted values, again, you'd be making a prediction interval that's too big..

It's better to understand why the assumptions are there.

share|improve this answer
add comment

As an aside, the four assumptions the OP mentions can easily be remembered using the acronym LINE:

  • Linearity
  • Independence
  • Normality
  • Equal Variance
share|improve this answer
add comment

The following are the assumptions of Linear Regression analysis.

Correct specification. The linear functional form is correctly specified.

Strict exogeneity. The errors in the regression should have conditional mean zero.

No multicollinearity. The regressors in X must all be linearly independent.

Homoscedasticity which means that the error term has the same variance in each observation.

No autocorrelation: the errors are uncorrelated between observations.

Normality. It is sometimes additionally assumed that the errors have normal distribution conditional on the regressors.

I.i.d observations: $(x_i, y_i)$ is independent from, and has the same distribution as, $(x_j, y_j)$ for all $i\neq j$.

For more information visit this page.

share|improve this answer
3  
Rather than "no multicolinearity" I'd say "no linear dependence". Collinearity is often used as a continuous rather than categorical measure. It is only strict or exact collinearity that is forbidden. –  Peter Flom Oct 3 '11 at 11:55
    
What about time series regression? What about generalised least squares? Your list reads a bit like list of commandments when in fact last 4 assumptions can be too restrictive if we only care about consistency and asymptotic normality of least squares estimate. –  mpiktas Oct 3 '11 at 12:41
    
Multicollinearity raises problems of interpretation (related to identifiability of some parameters) but it definitely is not a standard assumption of linear regression models. Near multicollinearity is primarily a computational problem but also raises similar issues of interpretation. –  whuber Oct 3 '11 at 14:37
    
@whuber & Peter Flom: As I read in the book of Gujarati at page no. 65-75. tiny.cc/cwb2g It count the "no multicollinearity" as a assumption of regression analysis. –  love-stats Oct 3 '11 at 16:59
    
@mpiktas: If you visit the given URL in the answer then you will find assumption about time series regression. –  love-stats Oct 3 '11 at 17:00
show 2 more comments

The assumption of linearity is that the model is linear in the parameters. It is fine to have a regression model with quadratic or higher order effects as long as the power function of the independent variable is part of a linear additive model. If the model does not contain higher order terms when it should, then the lack of fit will be evident in the plot of the residuals. However, standard regression models do not incorporate models in which the independent variable is raised to the power of a parameter (although there are other approaches that can be used to evaluate such models). Such models contain non-linear parameters.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.