3
$\begingroup$

Question

  1. Is there difference between assumptions of regression and assumptions of Ordinary Least Squares (OLS)?
  2. To apply OLS (the only regression method that I can use) on data, should my data and OLS meet both assumptions (assumptions of regression and assumptions of OLS)?
  3. Since 'non-linear' regression is also one kind of a regression, if it meets the assumptions of regression, can I apply non-linear regression? No matter whether it is a linear regression or non-linear regression, if the "assumptions of regression" are met, can I use either of them? Or Should there be another set of assumptions for non-linear regression?

More Details

  • I am studying linear regression, covering Ordinary Least Squares(OLS) so far.

  • I have learned the assumptions of regressions which should not be violated. The number of assumptions varies from book to book, but let me refer to that of the popular statistics youtuber "zedstatistics". The 6 assumptions of regression are as follows:

    1. Linearity
    2. Constant Error Variance
    3. Independent Error Terms
    4. Normal Errors
    5. No multi-collinearity between predictors
    6. Exogeneity

enter image description here

  • I thought if my data and model meets the six assumptions mentioned above, I could apply OLS on the data freely. However, I found on the Internet "assumptions of OLS" on the famous book website 'Econometrics with R'

OLS performs well under a quite broad variety of different circumstances. However, there are some assumptions which need to be satisfied in order to ensure that the estimates are normally distributed in large samples.

  1. The Error Term has Conditional Mean of Zero
  2. Independently and Identically Distributed Data
  3. Large Outliers are Unlikely
  • Now I am puzzled because it seems like there are two streams of assumptions to be met, in order to use OLS. The 1st assumption is the 'assumptions of regression (the 6 assumptions)' and we also need to check if the 2nd assumption (the 3 assumptions mentioned right above) is met as well.
  • Am I correct? Or are the two streams of assumptions saying the same thing, but expressed in a different way?
$\endgroup$
4
  • 2
    $\begingroup$ I am flattered to read that our book is referred to as "famous" :-). It should be noted that all its substantive content just follows the textbook by Stock and Watson, though. I think your question is somewhat too broad, though. Our 1 is 6 in the list above, 2 is related to 2&3, 3 is related to 2,3,4 but weaker (at the cost of less strong results), their 1 and 5 are stated elsewhere in Stock/Watson $\endgroup$ Oct 19, 2021 at 9:31
  • $\begingroup$ @ChristophHanck I love you guys' book :) One more question, though! Since 'non-linear' regression is also one kind of a regression, if it meets the assumptions of regression, can I apply non-linear regression? No matter whether it is a linear regression or non-linear regression, if the "assumptions of regression" are met, can I use either of them? Or Should there be another set of assumptions for non-linear regression? $\endgroup$
    – Eiffelbear
    Oct 19, 2021 at 10:46
  • $\begingroup$ Thanks :-). Indeed, there are also conditions under which nonlinear least squares or other techniques "work" for nonlinear models. These are a bit more complicated though, as such nonlinear approaches often do not have closed form solutions. See, e.g., the textbook by Hayashi (Econometrics) on extremum estimators. $\endgroup$ Oct 19, 2021 at 12:13
  • 1
    $\begingroup$ Many of the assumptions in your list are unnecessary. Normal errors and non-multicollinearity are prominent among the superfluous ones. Regardless, they are not "assumptions of regression:" they are assumptions made by some types of linear regression procedures. Among them, only linearity (as a function of the explanatory variables) is a universal assumption. $\endgroup$
    – whuber
    Oct 19, 2021 at 16:41

3 Answers 3

1
$\begingroup$

The OLS assumptions are incomplete, since they are missing the constant error variance and lack of multi-collinearity.

Moreover, the assumptions of OLS follow from the six assumptions listed by zedstatistics:

  1. Error term with conditional mean of zero, means the same as exogenity
  2. IID follows from the independent errors (independence) and from the normal errors (identically distributed).
  3. Large outliers are unlikely, follows from the normally distributed errors, as the tails of the normal distribution decay quickly.
$\endgroup$
2
  • $\begingroup$ Since 'non-linear' regression is also one kind of a regression, if it meets the assumptions of regression, can I apply non-linear regression? No matter whether it is a linear regression or non-linear regression, if the "assumptions of regression" are met, can I use either of them? Or Should there be another set of assumptions for non-linear regression? $\endgroup$
    – Eiffelbear
    Oct 19, 2021 at 10:47
  • $\begingroup$ I guess there is no single answer to this question, as it really depends on your non-linear model and on the type of inference you want to make. $\endgroup$
    – drmaettu
    Oct 19, 2021 at 12:05
1
$\begingroup$

The assumptions for the regression model are model assumptions to specify the behaviour in the model. By contrast, use of the OLS estimator is based on specification of the estimation method (which is not so much an assumption as a decision of how you want to do your estimation). You can either directly specify that you want to use OLS as your estimation method, or you can specify use of the MLE and this reduces down to OLS in the Gaussian regression model, or you can specify another estimator entirely.

$\endgroup$
0
$\begingroup$

There are the following 5 standard assumptions:

  • A0 (Linearity): The data generating process is given by $y_i = x_i\cdot\beta + u$.

Here $y$ is the dependent variable and $x$ is a vector of $k$ explanatory variables. The parameter $\beta$ determines the effect of the explanatory variables on the dependent variable. Deviations from this relationship are captured by the error term $u$.

  • A1 (No perfect multikolinearity): None of the $k$ explanatory variable is a perfect linear combination of the other.

This assumption ensures that the OLSE exists. If one variable was a perfect linear combination of the other (i.e. we can write one variable $a$ as $a = \lambda b + \delta c$, where $\lambda$ and $\delta$ are real numbers and $b$ and $c$ are other explanatory variables), it would be impossible for a linear estimator to distinguish between the three variables. Heuristically, the variables $a,b$ and $c$ would have to be estimated but since the variables are a perfect linear combination, there are only two independent equations. That is, we would have to solve for three variables with only two equations.

  • A2 (Mean independence): The conditional mean of $u$ given the explanatory variables $x_1, x_2, \dots, x_n$ in the sample is zero, i.e. $\mathbb E[u\mid x_1, x_2, \dots, x_n] = 0$.

This assumption ensures that the (conditional and unconditional!) mean of the OLSE is given by $\beta$, i.e. the OLSE is (conditionally and unconditionally) unbiased. Consequently, the average effect of the explanatory variables is given by $\hat\beta$.

  • A3 (Constant variance): The conditional mean of $u^2$ given the explanatory variables $x_1, x_2, \dots, x_n$ in the sample is a non-zero finite constant, i.e. $\mathbb E[u^2\mid x_1, x_2, \dots, x_n] = \sigma^2$ for some finite $\sigma > 0$.

This assumption ensures that the variance of the OLSE does not vary with the explanatory variables and that $\sigma^2$ can be easily estimated with the usual variance estimator $(n-k)^{-1}\sum_{i=1}^n(y_i - x_i\cdot\hat\beta)^2$, provided that assumption A2 holds. Furthermore, the (conditional and unconditional!) variance of the OLSE is then given by $\sigma^2(\sum_{i=1}^n x_ix_i')^{-1}$

  • A4 (Normality): The conditional distribution of $u$ given the explanatory variables $x_1, x_2, \dots, x_n$ in the sample is normal with mean $0$ and variance $\sigma^2$.

This assumption allows statistical inference in finite samples since the OLSE $\hat\beta$ will be also normal with mean $\beta$ and variance-covariance matrix $\sigma^2(\sum_{i=1}^n x_ix_i')^{-1}$. Assuming $\sigma$ is known, one could test whether the $\ell$th component of $\beta$ is equal to zero by checking whether $$\frac{e_\ell\cdot\beta}{\sqrt{\sigma^2e_\ell\cdot(\sum_{i=1}^n x_ix_i')^{-1}e_\ell}}$$ is larger than 1.96. Here $e_\ell$ denotes the $\ell$th basis canonical basis vector of the Euclidean space.

Another important consequence of assumption A4 is that $\hat\beta$ is not only BLUE (best linear unbiased estimator) but BUE (best unbiased estimator).


Note that we don't need the assumption that $u_1, u_2, \dots, u_n$ are independent and identically distributed (assumption A4 implies it in the above setting though). If we added the assumption, it would suffice to condition on $x_i$ instead of the full sample, as the full sample would be, by assumption independent, i.e. we would have, for example, $\mathbb E[u\mid x_1, x_2,\dots,x_n] = \mathbb E[u\mid x_i]$.

The sampling assumption is also very specific to the use-case. For example, independent errors works well for most cross section applications, but it is a very strong assumption for time series or panel data analysis.


$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.