What are the usual assumptions for linear regression?

Do they include:

  1. a linear relationship between the independent and dependent variable
  2. independent errors
  3. normal distribution of errors
  4. homoscedasticity

Are there any others?

  • 3
    You can find a rather complete list in William Berry's little book on "Understanding Regression Assumptions": books.google.com/books/about/… – user5644 Oct 3 '11 at 9:43
  • 3
    While respondents have listed some good resources, it is a difficult question to answer in this format, and (many) books have been devoted solely to this topic. There is no cook book, nor should there be given the potential variety of situations that linear regression could encompass. – Andy W Oct 3 '11 at 12:49
  • 2
    Technically, (ordinary) linear regression is a model of the form $\mathbb{E}[Y_i] = \mathbf{X}_i \beta$, $Y_i$ iid. That simple mathematical statement encompasses all the assumptions. This leads me to think, @Andy W, that you may be interpreting the question more broadly, perhaps in the sense of the art and practice of regression. Your further thoughts about this might be useful here. – whuber Oct 3 '11 at 14:41
  • 1
    @Andy W I wasn't trying to suggest your interpretation was incorrect. Your comment suggested a way of thinking about the question that goes beyond technical assumptions, perhaps pointing towards what may be needed for valid interpretation of regression results. It wouldn't be necessary to write a treatise in response, but even a list of some of those broader issues could be illuminating and might expand the scope and interest of this thread. – whuber Oct 3 '11 at 18:44
  • 1
    @whuber, if $EY_i=X_i\beta$ this means that the means are different for different $i$, hence $Y_i$ cannot be iid :) – mpiktas Oct 4 '11 at 7:35

The answer depends heavily on how do you define complete and usual. Suppose we write linear regression model in the following way:

$$y_i=\mathbf{x}_i'\beta+u_i$$

where $\mathbf{x}_i$ is the vector of predictor variables, $\beta$ is the parameter of interest, $y_i$ is the response variable, and $u_i$ are the disturbance. One of the possible estimates of $\beta$ is the least squares estimate:

$$\hat\beta=\textrm{argmin}_{\beta}\sum(y_i-\mathbf{x}_i\beta)^2=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum \mathbf{x}_iy_i$$

Now practically all of the textbooks deal with the assumptions when this estimate $\hat\beta$ has desirable properties, such as unbiasedness, consistency, efficiency, some distributional properties, etc.

Each of these properties requires certain assumptions, which are not the same. So the better question would be to ask which assumptions are needed for wanted properties of the LS estimate.

The properties I mention above require some probability model for regression. And here we have the situation where different models are used in different applied fields.

The simple case is to treat $y_i$ as an independent random variables, with $\mathbf{x}_i$ being non-random. I do not like the word usual, but we can say that this is the usual case in most applied fields (as far as I know).

Here is the list of some of the desirable properties of statistical estimates:

  1. The estimate exists
  2. Unbiasedness: $E\hat\beta=\beta$
  3. Consistency: $\hat\beta\to \beta$ as $n\to\infty$ ($n$ here is the size of a data sample)
  4. Efficiency: $Var(\hat\beta)$ is smaller than $Var(\tilde\beta)$ for alternative estimates $\tilde\beta$ of $\beta$
  5. The ability to either approximate or calculate the distribution function of $\hat\beta$

Existence

Existence property might seem weird, but it is very important. In the definition of $\hat\beta$ we invert the matrix

$$\sum \mathbf{x}_i\mathbf{x}_i'$$

It is not guaranteed that the inverse of this matrix exists for all possible variants of $\mathbf{x}_i$. So we immediately get our first assumption:

Matrix $$\sum \mathbf{x}_i\mathbf{x}_i'$$ should be of full rank, i.e. invertible.

Unbiasedness

We have

$$E\hat\beta=\left(\sum \mathbf{x}_i\mathbf{x}_i\right)^{-1}\left(\sum \mathbf{x}_iEy_i\right)=\beta,$$

if

$$Ey_i=\mathbf{x}_i\beta.$$

We may number it the second assumption, but we may have stated it outright, since this is one of the natural ways to define linear relationship.

Note that to get unbiasedness we only require that $Ey_i=\mathbf{x}_i\beta$ for all $i$, and $\mathbf{x_i}$ are constants. Independence property is not required.

Consistency

For getting the assumptions for consistency we need to state more clearly what do we mean by $\to$. For sequences of random variables we have different modes of convergence: in probability, almost surely, in distribution and p-th moment sense. Suppose we want to get the convergence in probability. We can use either law of large numbers or directly the Chebyshev inequality (we employ the fact that $E\hat\beta=\beta$):

$$P(|\hat\beta-\beta|>\varepsilon)\le \frac{Var(\beta)}{\varepsilon^2}$$

Since convergence in probability means that the left hand term must vanish for any $\varepsilon>0$ as $n\to\infty$, we need that $Var(\beta)\to 0$ as $n\to\infty$. This is perfectly reasonable since with more data the precision with which we estimate $\beta$ should increase.

We have that

$$Var(\hat\beta)=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_i\sum_j\mathbf{x}_i\mathbf{x}_j'cov(y_i,y_j)\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}$$

Independence ensures that $cov(y_i,y_j)=0$, hence the expression simplifies to

$$Var(\hat\beta)=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\sum_i\mathbf{x}_i\mathbf{x}_i'var(y_i)\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}$$

Now assume $Var(y_i)=const$, then

$$Var(\hat\beta)=\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}Var(y_i)$$

Now if we additionaly require that $\frac{1}{n}\sum \mathbf{x}_i\mathbf{x}_i'$ is bounded for each $n$ we immediately get

$$Var(\beta)\to 0, n\to\infty.$$

So to get the consistency we assumed that there is no autocorrelation ($cov(y_i,y_j)=0$), t the variance $Var(y_i)$ is constant and $\mathbf{x}_i$ do not grow too much. The first assumption is satisfied if $y_i$ comes from independent sample.

Efficiency

The classic result is the Gauss-Markov theorem. The conditions for it is exactly the first two conditions for consistency and the condition for unbiasedness.

Distributional properties

If $y_i$ are normal we immediately get that $\hat\beta$ is normal since it is a linear combination of normal random variables. If we assume previous assumptions of independence, uncorrelatedness and constant variance we get that

$$\hat\beta\sim N\left(\beta,\sigma^2\left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}\right)$$

where $Var(y_i)=\sigma^2$.

If $y_i$ are not normal, but independent we can get approximate distribution of $\hat\beta$ thanks to central limit theorem. For this we need to assume that

$$\lim_n \frac{1}{n}\sum\mathbf{x}_i\mathbf{x}_i'\to A$$

for some matrix $A$. The constant variance for assymptotic normality is not required if we assume that

$$\lim_n\frac{1}{n}\sum\mathbf{x}_i\mathbf{x}_i'Var(y_i)\to B$$

Note that with constant variance of $y$ we cant that $B=\sigma^2A$. The central limit theorem then gives us the following result:

$$\sqrt{n}(\hat\beta-\beta)\to N(0,A^{-1}BA^{-1}).$$

So from this we see that independence and constant variance for $y_i$ and certain assumptions for $\mathbf{x}_i$ gives us a lot of useful properties for LS estimate $\hat\beta$.

The thing is that these assumptions can be relaxed. For example we required that $\mathbf{x}_i$ are not random variables. This assumption is not feasible in econometric applications. If we let $\mathbf{x}_i$ be random, we can get similar results if use conditional expectations and take into account the randomness of $\mathbf{x}_i$. The independence assumption also can be relaxed. We already demonstrated that sometimes only uncorrelatedness is needed. Even this can be further relaxed and it is still possible to show that the LS estimate will be consistent and assymptoticaly normal. See for example White's book for more details.

  • This is a really clear, excellent answer!! – Patrick S. Forscher Dec 5 '12 at 20:33
  • A comment about the Gauss-Markov theorem. It only states that OLS is better than other estimators that are linear functions of the data. However, many commonly-used estimators, maximum likelihood (ML) in particular, are not linear functions of the data, and can be much more efficient than OLS under the conditions of the Gauss-Markov theorem. – Peter Westfall Jul 16 at 11:39
  • @PeterWestfall For gaussian normal errors, MLE is the OLS :) And you cannot get more efficient than MLE. I tried to be light with mathematical detail in this post. – mpiktas Jul 24 at 6:55
  • My point was that there are many more efficient estimators than OLS under non-normal distributions when the G-M conditions hold. G-M is essentially useless as a statement that OLS is "good" under non-normality, because the best estimators in non-normal cases are nonlinear functions of the data. – Peter Westfall Jul 25 at 13:32
  • @mpiktas So either we take $\mathbf x$ as not random, and use estimator $\mathbf{\hat{Y}}$ or we take $\mathbf x$ as random and use estimator $\mathbf{\hat{Y|x}}$? – Paari Vendhan 5 hours ago

There are a number of good answers here. It occurs to me that there is one assumption that has not been stated however (at least not explicitly). Specifically, a regression model assumes that $\mathbf X$ (the values of your explanatory / predictor variables) is fixed and known, and that all of the uncertainty in the situation exists within the $Y$ variable. In addition, this uncertainty is assumed to be sampling error only.

Here are two ways to think about this: If you are building an explanatory model (modeling experimental results), you know exactly what the levels of the independent variables are, because you manipulated / administered them. Moreover, you decided what those levels would be before you ever started gathering data. So you are conceptualizing all of the uncertainty in the relationship as existing within the response. On the other hand, if you are building a predictive model, it is true that the situation differs, but you still treat the predictors as though they were fixed and known, because, in the future, when you use the model to make a prediction about the likely value of $y$, you will have a vector, $\mathbf x$, and the model is designed to treat those values as though they are correct. That is, you will be conceiving of the uncertainty as being the unknown value of $y$.

These assumptions can be seen in the equation for a prototypical regression model: $$ y_i = \beta_0 + \beta_1x_i + \varepsilon_i $$ A model with uncertainty (perhaps due to measurement error) in $x$ as well might have the same data generating process, but the model that's estimated would look like this: $$ y_i = \hat\beta_0 + \hat\beta_1(x_i + \eta_i) + \hat\varepsilon_i, $$ where $\eta$ represents random measurement error. (Situations like the latter have led to work on errors in variables models; a basic result is that if there is measurement error in $x$, the naive $\hat\beta_1$ would be attenuated--closer to 0 than its true value, and that if there is measurement error in $y$, statistical tests of the $\hat\beta$'s would be underpowered, but otherwise unbiased.)

One practical consequence of the asymmetry intrinsic in the typical assumption is that regressing $y$ on $x$ is different from regressing $x$ on $y$. (See my answer here: What is the difference between doing linear regression on y with x versus x with y? for a more detailed discussion of this fact.)

  • What does it mean "fixed" | "random" in plain language? And how to distinguish between fixed and random effects(=factors)? I think that in my design there is 1 fixed known factor with 5 levels. Right? – stan Dec 9 '12 at 10:22
  • @stan, I recognize your confusion. Terminology in stats is often confusing & unhelpful. In this case, "fixed" is not quite the same as the fixed in 'fixed effects & random effects' (although they are related). Here, we're not talking about effects--we're talking about the $X$ data, ie your predictor / explanatory variables. The easiest way to understand the idea of your $X$ data being fixed is to think of a planned experiment. Before you have done anything, when you're designing the experiment, you decide what the levels of your explanatory will be, you don't discover them along the way. – gung Dec 9 '12 at 16:00
  • W/ predictive modeling, that's not quite true, but we will treat our $X$ data that way in the future, when we use the model to make predictions. – gung Dec 9 '12 at 16:00
  • Why do the βs and the ε have a hat in the bottom equation, but not in the top one? – user1205901 Apr 14 '15 at 8:41
  • 2
    @user1205901, the top model is of the data generating process, the bottom is your estimate of it. – gung Apr 14 '15 at 14:13

As an aside, the four assumptions the OP mentions can easily be remembered using the acronym LINE:

  • Linearity
  • Independence
  • Normality
  • Equal Variance

Different assumptions can be used to justify OLS

  • In some situations, an author tests the residuals for normality.
    • But in other situations, the residuals aren't normal and the author uses OLS anyway!
  • You'll see texts saying that homoscedasticity is an assumption.
    • But you see researchers using OLS when homoscedasticity is violated.

What gives?!

An answer is that somewhat different sets of assumptions can be used to justify the use of ordinary least squares (OLS) estimation. OLS is a tool like a hammer: you can use a hammer on nails but you can also use it on pegs, to break apart ice, etc...

Two broad categories of assumptions are those that apply to small samples and those that rely on large samples so that the central limit theorem can be applied.

1. Small sample assumptions

Small sample assumptions as discussed in Hayashi (2000) are:

  1. Linearity
  2. Strict exogeneity
  3. No multicollinearity
  4. Spherical errors (homoscedasticity)

Under (1)-(4), the Gauss-Markov theorem applies, and the ordinary least squares estimator is the best linear unbiased estimator.

  1. Normality of error terms

Further assuming normal error terms allows hypothesis testing. If the error terms are conditionally normal, the distribution of the OLS estimator is also conditionally normal.

Another noteworthy point is that with normality, the OLS estimator is also the maximum likelihood estimator.

2. Large sample assumptions

These assumptions can be modified/relaxed if we have a large enough sample so that we can lean on the law of large numbers (for consistency of the OLS estimator) and the central limit theorem (so that the sampling distribution of the OLS estimator converges to the normal distribution and we can do hypothesis testing, talk about p-values etc...).

Hayashi is a macroeconomics guy and his large sample assumptions are formulated with the time series context in mind:

  1. linearity
  2. ergodic stationarity
  3. predetermined regressors: error-terms are orthogonal to their contemporaneous error terms.
  4. $\operatorname{E}[\mathbf{x}\mathbf{x}']$ is full rank
  5. $\mathbf{x}_i \epsilon_i$ is a martingale difference sequence with finite second moments.
  6. Finite 4th moments of regressors

You may encounter stronger versions of these assumptions, for example, that error terms are independent.

Proper large sample assumptions get you to a sampling distribution of the OLS estimator that is asymptotically normal.

References

Hayashi, Fumio, 2000, Econometrics

It's all about what you wanna do with your model. Imagine if your errors were all positively skewed/non-normal. If you wanted to make a prediction interval, you could do better than using the t-distribution.. If your variance is smaller at smaller predicted values, again, you'd be making a prediction interval that's too big..

It's better to understand why the assumptions are there.

The following are the assumptions of Linear Regression analysis.

Correct specification. The linear functional form is correctly specified.

Strict exogeneity. The errors in the regression should have conditional mean zero.

No multicollinearity. The regressors in X must all be linearly independent.

Homoscedasticity which means that the error term has the same variance in each observation.

No autocorrelation: the errors are uncorrelated between observations.

Normality. It is sometimes additionally assumed that the errors have normal distribution conditional on the regressors.

I.i.d observations: $(x_i, y_i)$ is independent from, and has the same distribution as, $(x_j, y_j)$ for all $i\neq j$.

For more information visit this page.

  • 4
    Rather than "no multicolinearity" I'd say "no linear dependence". Collinearity is often used as a continuous rather than categorical measure. It is only strict or exact collinearity that is forbidden. – Peter Flom Oct 3 '11 at 11:55
  • 2
    What about time series regression? What about generalised least squares? Your list reads a bit like list of commandments when in fact last 4 assumptions can be too restrictive if we only care about consistency and asymptotic normality of least squares estimate. – mpiktas Oct 3 '11 at 12:41
  • 1
    Multicollinearity raises problems of interpretation (related to identifiability of some parameters) but it definitely is not a standard assumption of linear regression models. Near multicollinearity is primarily a computational problem but also raises similar issues of interpretation. – whuber Oct 3 '11 at 14:37
  • @whuber & Peter Flom: As I read in the book of Gujarati at page no. 65-75. tiny.cc/cwb2g It count the "no multicollinearity" as a assumption of regression analysis. – love-stats Oct 3 '11 at 16:59
  • @mpiktas: If you visit the given URL in the answer then you will find assumption about time series regression. – love-stats Oct 3 '11 at 17:00

The assumption of linearity is that the model is linear in the parameters. It is fine to have a regression model with quadratic or higher order effects as long as the power function of the independent variable is part of a linear additive model. If the model does not contain higher order terms when it should, then the lack of fit will be evident in the plot of the residuals. However, standard regression models do not incorporate models in which the independent variable is raised to the power of a parameter (although there are other approaches that can be used to evaluate such models). Such models contain non-linear parameters.

There is no such a thing as a single list of assumptions, there will be at least 2: one for fixed and one for random design matrix. Plus you may want to look at the assumptions for time series regressions (see p.13)

The case when the design matrix $X$ is fixed could be the most common one, and its assumptions are often expressed as a Gauss-Markov theorem. The fixed design means that you truly control the regressors. For instance, you conduct an experiment and can set the parameters such as temperature, pressure etc. See also p.13 here.

Unfortunately, in social sciences such as economics you rarely can control the parameters of the experiment. Usually, you observe what happens in economy, record the environment metrics, then regress on them. It turns out that it's a very different and more difficult situation, called a random design. In this case the Gauss-Markov theorem is modified also see p.12 here. You can see how the conditions are now expressed in terms of conditional probabilities, which is not an innocuous change.

In econometrics the assumptions have names:

  • linearity
  • strict exogeneity
  • no multicollinearity
  • spherical error variance (includes homoscedasticity and no correlation)

Notice that I never mentioned normality. It's not a standard assumption. It's often used in intro regression courses because it makes some derivations easier, but it's not required for regression to work and have nice properties.

The least squares regression coefficient provides a way to summarize the first order trend in any kind of data. @mpiktas answer is a thorough treatment of the conditions under which least squares is increasingly optimal. I'd like to go the other way and show the most general case when least squares works. Let's see the most general formulation of the least-squares equation:

$$E[Y|X] = \alpha + \beta X$$

It's just a linear model for the conditional mean of the response.

Note I've bucked the error term. If you'd like to summarize the uncertainty of $\beta$, then you must appeal to the central limit theorem. The most general class of least squares estimators converge to normal when the Lindeberg condition is met: boiled down, the Lindeberg condition for least squares requires that the fraction of the largest squared residual to the sum of the sum of squared residuals must go to 0 as $n \rightarrow \infty$. If your design will keep sampling larger and larger residuals, then the experiment is "dead in the water".

When the Lindeberg condition is met, the regression parameter $\beta$ is well defined, and the estimator $\hat{\beta}$ is an unbiased estimator that has a known approximating distribution. More efficient estimators may exist. In other cases of heteroscedasticity, or correlated data, usually a weighted estimator is more efficient. That's why I would never advocate using the naïve methods when better ones are available. But they often are not!

  • For the econometricians: It is worth pointing out that this condition implies strict exogeneity, so strict exogeneity need not be stated as an assumption in the conditional mean model. It is automatically true, mathematically. (Talking theory here, not estimates.) – Peter Westfall Oct 25 at 18:19

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.