What is a complete list of the usual assumptions for linear regression?

Question

What are the usual assumptions for linear regression?

Do they include:

1. a linear relationship between the independent and dependent variable
2. independent errors
3. normal distribution of errors
4. homoscedasticity

Are there any others?

Answer 1

The answer depends heavily on how do you define complete and usual. Suppose we write linear regression model in the following way:$ \newcommand{\x}{\mathbf{x}} \newcommand{\bet}{\boldsymbol\beta} \DeclareMathOperator{\E}{\mathbb{E}} \DeclareMathOperator{\Var}{Var} \DeclareMathOperator{\Cov}{Cov} \DeclareMathOperator{\Tr}{Tr} $

$$y_i = \x_i'\bet + u_i$$

where $\mathbf{x}_i$ is the vector of predictor variables, $\beta$ is the parameter of interest, $y_i$ is the response variable, and $u_i$ are the disturbance. One of the possible estimates of $\beta$ is the least squares estimate: $$ \hat\bet = \textrm{argmin}_{\bet}\sum(y_i-\x_i\bet)^2 = \left(\sum \x_i \x_i'\right)^{-1} \sum \x_i y_i .$$

Now practically all of the textbooks deal with the assumptions when this estimate $\hat\bet$ has desirable properties, such as unbiasedness, consistency, efficiency, some distributional properties, etc.

Each of these properties requires certain assumptions, which are not the same. So the better question would be to ask which assumptions are needed for wanted properties of the LS estimate.

The properties I mention above require some probability model for regression. And here we have the situation where different models are used in different applied fields.

The simple case is to treat $y_i$ as an independent random variables, with $\x_i$ being non-random. I do not like the word usual, but we can say that this is the usual case in most applied fields (as far as I know).

Here is the list of some of the desirable properties of statistical estimates:

1. The estimate exists.
2. Unbiasedness: $E\hat\bet=\bet$.
3. Consistency: $\hat\bet \to \bet$ as $n\to\infty$ ($n$ here is the size of a data sample).
4. Efficiency: $\Var(\hat\bet)$ is smaller than $\Var(\tilde\bet)$ for alternative estimates $\tilde\bet$ of $\bet$.
5. The ability to either approximate or calculate the distribution function of $\hat\bet$.

Existence

Existence property might seem weird, but it is very important. In the definition of $\hat\beta$ we invert the matrix $\sum \x_i \x_i'.$

It is not guaranteed that the inverse of this matrix exists for all possible variants of $\x_i$. So we immediately get our first assumption:

Matrix $\sum \x_i \x_i'$ should be of full rank, i.e. invertible.

Unbiasedness

We have $$ \E\hat\bet = \left(\sum \x_i \x_i' \right)^{-1}\left(\sum \x_i \E y_i \right) = \bet, $$ if $$\E y_i = \x_i \bet.$$

We may number it the second assumption, but we may have stated it outright, since this is one of the natural ways to define linear relationship.

Note that to get unbiasedness we only require that $\E y_i = \x_i \bet$ for all $i$, and $\x_i$ are constants. Independence property is not required.

Consistency

For getting the assumptions for consistency we need to state more clearly what do we mean by $\to$. For sequences of random variables we have different modes of convergence: in probability, almost surely, in distribution and $p$-th moment sense. Suppose we want to get the convergence in probability. We can use either law of large numbers, or directly use the multivariate Chebyshev inequality (employing the fact that $\E \hat\bet = \bet$):

$$\Pr(\lVert \hat\bet - \bet \rVert >\varepsilon)\le \frac{\Tr(\Var(\hat\bet))}{\varepsilon^2}.$$

(This variant of the inequality comes directly from applying Markov's inequality to $\lVert \hat\bet - \bet\rVert^2$, noting that $\E \lVert \hat\bet - \bet\rVert^2 = \Tr \Var(\hat\bet)$.)

Since convergence in probability means that the left hand term must vanish for any $\varepsilon>0$ as $n\to\infty$, we need that $\Var(\hat\bet)\to 0$ as $n\to\infty$. This is perfectly reasonable since with more data the precision with which we estimate $\bet$ should increase.

We have that $$ \Var(\hat\bet) =\left( \sum \x_i \x_i' \right)^{-1} \left( \sum_i \sum_j \x_i \x_j' \Cov(y_i, y_j) \right) \left(\sum \mathbf{x}_i\mathbf{x}_i'\right)^{-1}.$$

Independence ensures that $\Cov(y_i, y_j) = 0$, hence the expression simplifies to $$ \Var(\hat\bet) = \left( \sum \x_i \x_i' \right)^{-1} \left( \sum_i \x_i \x_i' \Var(y_i) \right) \left( \sum \x_i \x_i' \right)^{-1} .$$

Now assume $\Var(y_i) = \text{const}$, then $$ \Var(\hat\beta) = \left(\sum \x_i \x_i' \right)^{-1} \Var(y_i) .$$

Now if we additionally require that $\frac{1}{n} \sum \x_i \x_i'$ is bounded for each $n$, we immediately get $$\Var(\bet) \to 0 \text{ as } n \to \infty.$$

So to get the consistency we assumed that there is no autocorrelation ($\Cov(y_i, y_j) = 0$), the variance $\Var(y_i)$ is constant, and the $\x_i$ do not grow too much. The first assumption is satisfied if $y_i$ comes from independent samples.

Efficiency

The classic result is the Gauss-Markov theorem. The conditions for it is exactly the first two conditions for consistency and the condition for unbiasedness.

Distributional properties

If $y_i$ are normal we immediately get that $\hat\bet$ is normal, since it is a linear combination of normal random variables. If we assume previous assumptions of independence, uncorrelatedness and constant variance we get that $$ \hat\bet \sim \mathcal{N}\left(\bet, \sigma^2\left(\sum \x_i \x_i' \right)^{-1} \right)$$ where $\Var(y_i)=\sigma^2$.

If $y_i$ are not normal, but independent, we can get approximate distribution of $\hat\bet$ thanks to the central limit theorem. For this we need to assume that $$\lim_{n \to \infty} \frac{1}{n} \sum \x_i \x_i' \to A$$ for some matrix $A$. The constant variance for asymptotic normality is not required if we assume that $$\lim_{n \to \infty} \frac{1}{n} \sum \x_i \x_i' \Var(y_i) \to B.$$

Note that with constant variance of $y$, we have that $B = \sigma^2 A$. The central limit theorem then gives us the following result:

$$\sqrt{n}(\hat\bet - \bet) \to \mathcal{N}\left(0, A^{-1} B A^{-1} \right).$$

So from this we see that independence and constant variance for $y_i$ and certain assumptions for $\mathbf{x}_i$ gives us a lot of useful properties for LS estimate $\hat\bet$.

The thing is that these assumptions can be relaxed. For example we required that $\x_i$ are not random variables. This assumption is not feasible in econometric applications. If we let $\x_i$ be random, we can get similar results if use conditional expectations and take into account the randomness of $\x_i$. The independence assumption also can be relaxed. We already demonstrated that sometimes only uncorrelatedness is needed. Even this can be further relaxed and it is still possible to show that the LS estimate will be consistent and asymptoticaly normal. See for example White's book for more details.

Answer 2

There are a number of good answers here. It occurs to me that there is one assumption that has not been stated however (at least not explicitly). Specifically, a regression model assumes that $\mathbf X$ (the values of your explanatory / predictor variables) is fixed and known, and that all of the uncertainty in the situation exists within the $Y$ variable. In addition, this uncertainty is assumed to be sampling error only.

Here are two ways to think about this: If you are building an explanatory model (modeling experimental results), you know exactly what the levels of the independent variables are, because you manipulated / administered them. Moreover, you decided what those levels would be before you ever started gathering data. So you are conceptualizing all of the uncertainty in the relationship as existing within the response. On the other hand, if you are building a predictive model, it is true that the situation differs, but you still treat the predictors as though they were fixed and known, because, in the future, when you use the model to make a prediction about the likely value of $y$, you will have a vector, $\mathbf x$, and the model is designed to treat those values as though they are correct. That is, you will be conceiving of the uncertainty as being the unknown value of $y$.

These assumptions can be seen in the equation for a prototypical regression model: $$ y_i = \beta_0 + \beta_1x_i + \varepsilon_i $$ A model with uncertainty (perhaps due to measurement error) in $x$ as well might have the same data generating process, but the model that's estimated would look like this: $$ y_i = \hat\beta_0 + \hat\beta_1(x_i + \eta_i) + \hat\varepsilon_i, $$ where $\eta$ represents random measurement error. (Situations like the latter have led to work on errors in variables models; a basic result is that if there is measurement error in $x$, the naive $\hat\beta_1$ would be attenuated--closer to 0 than its true value, and that if there is measurement error in $y$, statistical tests of the $\hat\beta$'s would be underpowered, but otherwise unbiased.)

One practical consequence of the asymmetry intrinsic in the typical assumption is that regressing $y$ on $x$ is different from regressing $x$ on $y$. (See my answer here: What is the difference between doing linear regression on y with x versus x with y? for a more detailed discussion of this fact.)

Answer 3

The following diagrams show which assumptions are required to get which implications in the finite and asymptotic scenarios.

Linear Regression Assumptions: Key Points

Generally the assumptions can be broken down into what we need for our coefficient estimators

1. to be right on average--unbiased--or right with infinite data--consistent and
2. to follow a certain distribution so we can know how precisely we are measuring them.

Unbiasedness / Consistency

We want our coefficients to be right on average (unbiased) or at least right if we have a lot of data (consistent).

If you want unbiased coefficients, the key assumption is strict exogeneity. This means that the average value of the error term in the regression is 0 given the covariates used in the regression.

For consistent coefficients, the key assumption is “predetermined regressors” which is implied by: "there is no correlation between the error term and any of the covariates of the regression" if a constant is included in the regression.

Strictly speaking, there is no way to confirm these assumptions are right without randomly assigning the covariate whose coefficient you want to get right. Without random assignment, you have to make a qualitative argument that the assumptions are met. However, if you make a scatter plot of residuals on the y axis and the predicted outcome value on the x axis and there is a systematic trend away from 0, that’s a sign this assumption (or the linearity assumption) is not met.

Assumptions are also important to understand the precision of coefficient estimates.

Understanding the Precision of the Coefficients

Homoskedasticity and normality are not needed for unbiased/consistent coefficients. You only need these additional assumptions if you want to get a sense of the precision with which you are measuring your coefficients with shortcut methods (e.g. F tests). However, you can always use heteroskedasticity robust standard errors, bootstrapping, or randomization inference to understand precision instead (descriptions and examples of these latter procedures can be found in my post here).

Answer 4

The assumptions of the classical linear regression model include:

1. Linear Parameter and correct model specification
2. Full Rank of the X Matrix
3. Explanatory Variables must be exogenous
4. Independent and Identically Distributed Error Terms
5. Normal Distributed Error Terms in Population

Although the answers here provide already a good overview of the classical OLS assumption, you can find a more comprehensive description of the assumption of the classical linear regression model here:

https://economictheoryblog.com/2015/04/01/ols_assumptions/

In addition, the article describes the consequences in case one violates certain assumptions.

Answer 5

Different assumptions can be used to justify OLS

• In some situations, an author tests the residuals for normality.
  • But in other situations, the residuals aren't normal and the author uses OLS anyway!
• You'll see texts saying that homoscedasticity is an assumption.
  • But you see researchers using OLS when homoscedasticity is violated.

What gives?!

An answer is that somewhat different sets of assumptions can be used to justify the use of ordinary least squares (OLS) estimation. OLS is a tool like a hammer: you can use a hammer on nails but you can also use it on pegs, to break apart ice, etc...

Two broad categories of assumptions are those that apply to small samples and those that rely on large samples so that the central limit theorem can be applied.

1. Small sample assumptions

Small sample assumptions as discussed in Hayashi (2000) are:

1. Linearity
2. Strict exogeneity
3. No multicollinearity
4. Spherical errors (homoscedasticity)

Under (1)-(4), the Gauss-Markov theorem applies, and the ordinary least squares estimator is the best linear unbiased estimator.

5. Normality of error terms

Further assuming normal error terms allows hypothesis testing. If the error terms are conditionally normal, the distribution of the OLS estimator is also conditionally normal.

Another noteworthy point is that with normality, the OLS estimator is also the maximum likelihood estimator.

2. Large sample assumptions

These assumptions can be modified/relaxed if we have a large enough sample so that we can lean on the law of large numbers (for consistency of the OLS estimator) and the central limit theorem (so that the sampling distribution of the OLS estimator converges to the normal distribution and we can do hypothesis testing, talk about p-values etc...).

Hayashi is a macroeconomics guy and his large sample assumptions are formulated with the time series context in mind:

1. linearity
2. ergodic stationarity
3. predetermined regressors: error-terms are orthogonal to their contemporaneous error terms.
4. $\operatorname{E}[\mathbf{x}\mathbf{x}']$ is full rank
5. $\mathbf{x}_i \epsilon_i$ is a martingale difference sequence with finite second moments.
6. Finite 4th moments of regressors

You may encounter stronger versions of these assumptions, for example, that error terms are independent.

Proper large sample assumptions get you to a sampling distribution of the OLS estimator that is asymptotically normal.

References

Hayashi, Fumio, 2000, Econometrics

Answer 6

It's all about what you want to do with your model. Imagine if your errors were positively skewed/non-normal. If you wanted to make a prediction interval, you could do better than using the t-distribution. If your variance is smaller at smaller predicted values, again, you'd be making a prediction interval that's too big.

It's better to understand why the assumptions are there.

Answer 7

The least squares regression coefficient provides a way to summarize the first order trend in any kind of data. @mpiktas answer is a thorough treatment of the conditions under which least squares is increasingly optimal. I'd like to go the other way and show the most general case when least squares works. Let's see the most general formulation of the least-squares equation:

$$E[Y|X] = \alpha + \beta X$$

It's just a linear model for the conditional mean of the response.

Note I've bucked the error term. If you'd like to summarize the uncertainty of $\beta$, then you must appeal to the central limit theorem. The most general class of least squares estimators converge to normal when the Lindeberg condition is met: boiled down, the Lindeberg condition for least squares requires that the fraction of the largest squared residual to the sum of the sum of squared residuals must go to 0 as $n \rightarrow \infty$. If your design will keep sampling larger and larger residuals, then the experiment is "dead in the water".

When the Lindeberg condition is met, the regression parameter $\beta$ is well defined, and the estimator $\hat{\beta}$ is an unbiased estimator that has a known approximating distribution. More efficient estimators may exist. In other cases of heteroscedasticity, or correlated data, usually a weighted estimator is more efficient. That's why I would never advocate using the naïve methods when better ones are available. But they often are not!

Answer 8

There is no such a thing as a single list of assumptions, there will be at least 2: one for fixed and one for random design matrix. Plus you may want to look at the assumptions for time series regressions (see p.13)

The case when the design matrix $X$ is fixed could be the most common one, and its assumptions are often expressed as a Gauss-Markov theorem. The fixed design means that you truly control the regressors. For instance, you conduct an experiment and can set the parameters such as temperature, pressure etc. See also p.13 here.

Unfortunately, in social sciences such as economics you rarely can control the parameters of the experiment. Usually, you observe what happens in economy, record the environment metrics, then regress on them. It turns out that it's a very different and more difficult situation, called a random design. In this case the Gauss-Markov theorem is modified also see p.12 here. You can see how the conditions are now expressed in terms of conditional probabilities, which is not an innocuous change.

In econometrics the assumptions have names:

• linearity
• strict exogeneity
• no multicollinearity
• spherical error variance (includes homoscedasticity and no correlation)

Notice that I never mentioned normality. It's not a standard assumption. It's often used in intro regression courses because it makes some derivations easier, but it's not required for regression to work and have nice properties.

Answer 9

The assumption of linearity is that the model is linear in the parameters. It is fine to have a regression model with quadratic or higher order effects as long as the power function of the independent variable is part of a linear additive model. If the model does not contain higher order terms when it should, then the lack of fit will be evident in the plot of the residuals. However, standard regression models do not incorporate models in which the independent variable is raised to the power of a parameter (although there are other approaches that can be used to evaluate such models). Such models contain non-linear parameters.

Answer 10

The following are the assumptions of Linear Regression analysis.

Correct specification. The linear functional form is correctly specified.

Strict exogeneity. The errors in the regression should have conditional mean zero.

No multicollinearity. The regressors in X must all be linearly independent.

Homoscedasticity which means that the error term has the same variance in each observation.

No autocorrelation: the errors are uncorrelated between observations.

Normality. It is sometimes additionally assumed that the errors have normal distribution conditional on the regressors.

i.i.d observations: $(x_i, y_i)$ is independent from, and has the same distribution as, $(x_j, y_j)$ for all $i\neq j$.

For more information visit the Wiki page on same.