Understanding the assumptions of Linear Regression

Question

I have been studying the four assumptions of linear regression and different sources give different interpretations of the same.

The four assumptions are :

a) Linearity - Existence of linear relationship between the dependent and the independent variable

b) Normality - The residual values must be normally distributed ( But, some sources say that the Response variable must be normally distributed. I don't know which one is true)

c) Independence - The residuals must be independent of each other

d) Homoscedasticity - The residuals have constant variance for any value of X.

I have understood the first assumption , i.e, the Linearity. But, I'm finding it hard to understand "Why" the remaining three assumptions must hold true for a better model. I haven't found a proper reasoning for this anywhere. Even the standard textbooks do not go deep into this topic.

Can someone help me understand "why" the b), c) and d) assumptions must be true ? Also, I am not sure about the normality condition that if the response variable must be normal OR the residuals should be normally distributed. ( Different sources say different things).

Answer 1

Condition 1

You seem to understand this one. This assumption can be written as $f(x_i) = \beta^\mathrm{T}x_i$ and in matrix form as $\mathbf{f} = \mathbf{X}\beta$

Condition 2

In short, you need that $\mathbb{P}(y\mid x) = \mathcal{N}(\mu(x), \sigma^2)$ where $\mu(x)$ is a function of $X$. This assumption is met when the residuals are normal since $y = f(x) + \varepsilon$ and $\varepsilon\sim\mathcal{N}(0,\sigma^2)$ means that $y \mid x \sim \mathcal{N}(f(x),\sigma^2)$. So yes, you could say that

The response variable $y$ needs to follow a normal distribution

but most of the time it's because the residuals $\varepsilon$ are normal.

Condition 3

This condition is needed to justify the OLS loss which is $$\mathcal{L}(\hat{f}, \mathcal{D}) \propto \sum_{i=1}^n\,\lVert y_i-\hat{f}(x_i)\rVert^2 = \lVert \mathbf{y}-\hat{\mathbf{f}}\rVert^2 $$

If you want to avoid this condition you then need to change your loss to $(\mathbf{y}-\hat{\mathbf{f}})^\mathrm{T}\mathbf{W}(\mathbf{y}-\hat{\mathbf{f}})$ where $\mathbf{W} = \pmb{\Sigma}^{-1}$ is the covariance of the noise terms. This is called generalized least squares.

Condition 4

This condition means that the variance $\mathrm{Var}[Y\mid X=x]$ is constant. This is what is needed to have the loss $$\mathcal{L}(\hat{f}, \mathcal{D}) = \frac{1}{2\sigma^2}\sum_{i=1}^n\,\lVert y_i-\hat{f}(x_i)\rVert^2$$ If instead you want to model a noise variance which changes as $X$ takes different values, you get the loss $$\mathcal{L}(\hat{f}, \mathcal{D}) = \sum_{i=1}^n\,\frac{1}{2\sigma^2(x_i)}\lVert y_i-\hat{f}(x_i)\rVert^2 = \sum_{i=1}^n\,w(x_i)\lVert y_i-\hat{f}(x_i)\rVert^2$$ which weights each residuals according to a function $w(x_i)$ of the input value $x_i$. This is similar to LOESS regression.

Answer 2

Just to add to the discussion.

Assumption 1. Linearity

This assumption says that we believe the true population model is linear in parameters (not in explanatory variables). Thus: $$ y = a_0 + a_1 x + a_2 x^2 + a_3 ln(x) +e$$ Is a linear model. However, you can also interpret it in a slighly different way. the linearity assumption says that the conditional mean is a linear function of parameters: $$ E(y|x) = = a_0 + a_1 x + a_2 x^2 + a_3 ln(x) $$ Which is a bit more flexible, because we are basically averaging out the error.

This assumption basically imposes restrictions on how the error affects the model and allows us to estimate a Linear Regression model, usually via OLS. There are other important assumptions left tho.

1a) Random sampling: Your sample is representative of the population you want to study or say something about.

1b) the Var(X)>>0 and/or no multicollinearity. You want your data to have variation (which is used to identify coefficients in the model, but that variation has to be unique that that variable. For example, if $X1 = X2$, then you cannot include both in a model because when controlling for one, the other variable does not vary.

1c) you want X to be independent of the population unobservables $e$. Otherwise, your model and coefficients cannot be interpreted as causal relationships.

ADDED: Keep in mind that once you estimate your model, the errors $\hat{e}$ are by construction linearly independent with X. However, they are not the same as the population unobservables $e$.

Now why would this allow you to estimate causal relationships? Lets consider the assumptions again. Model is linear: $$y=a_0+a_1*x +e$$. If X is independent of e, and we can observe both, the causal effect of $x$ on $y$ could be done as follows: $$y^1=a_0+a_1*(x+1) +e$$ $$y^0=a_0+a_1*(x) +e$$ Thus the effect of a 1 unit change in $x$ can be is just $y^1-y^0$. This is possible because we can "assume" e is constant.

However, When X and e are correlated, if $X$ changes, then $e$ may change too for unknown reasons: $$y^1=a_0+a_1*(x+1) +e + \Delta e$$ $$y^0=a_0+a_1*(x) +e$$ In this case $y^1-y^0$ is not the effect of a change in X, because it also includes a change in $e$ we cannot explain. (technically we do not even know it is there)

When X and $e$ are correlated in the population model, and you estimate it via OLS, $\hat{a_1}$ will be a combination of $a_1$ and the correlation between $X$ and $e$.

The rest of the assumptions do not have to do with the estimation of the coefficient in a Linear regression, but with the estimation of the standard errors of those coefficients, and the efficiency of information use. Thus they are important, but you can live without them.

Assumption 2. Normality

This is not a "MUST". the LR regression model in fact imposes no assumption on the errors. your errors could be poisson, uniform, chi2, etc etc etc, and you could still estimate the LR using OLS. Now, if you want to estimate the model with other method, then yes, you need to impose a distributional assumption on the errors.

Then why is it added as an assumption in some texts?. One reason is as follows. When you estimate your LR using OLS, you find the following solution for $\beta s$: $$\hat{\beta}=(X'X)^{-1} X'Y$$ $$\hat{\beta}=(X'X)^{-1} X'(X\beta+e)$$ $$\hat{\beta}=\beta+(X'X)^{-1} X'e$$ So the estimated coefficients are a function of the errors $e$. Now, if you want to run tests on $\beta s$, you need to know something about their distribution (is it symmetrical, or asymmetrical, or flat, or all bundle up with a few extreme values). If the errors are normal, however, $\beta s$ are also normal, so we can use the family of Normalbased distributions (t-test, F-test, Chi2, z etc), to do inference.

In other words, the assumption of normality just makes life easier because it warrants the estimated coefficients are also normal.

What if $e$ is not normal? When the sample is large enough...(no strict criteria that i know to decide what is large enough), the normality of the errors are no longer needed, and one relies on the Central limit theory. This basically implies that $\beta$ still distributions as normal, even if the errors $e$ do not.

So, $e$ doesnt need to be normal, but its a good assumption that makes it easy to accept that $\hat{\beta}$ is normally distributed (and you can use standard tests).

Assumption 3. Independence

This assumption is usually taken as given. When you have crossection data, and you get a random sample of the population (truly random), it is easy to assume that those unobservables we leave in $e$ are independent of each other. When data is collected from "clusters", this may not be true.

For example, if you get info from families, it is very likely that some of those "errors" are common among all family members, although they are independent across different families.

When this happens, the estimation of standard errors of the $\beta s$ will be incorrect. The standard formulas assume errors are independent, but if they are indeed correlated, using standard formulas will most likely understate the true variation of the betas.

In terms of the LR. Independence of the errors is a property that simplifies the estimation of standard errors, but it is not necessary, since you can "correct" for it.

Assumption 4. Homoskedasticity

So this assumption means variation of $e$ does not change with X. In other words, every single observation has the same amount of information for the estimation of $\beta s$ because the error variation is constant.

When there is heteroskedasticity, this is no longer the case. $Var(e|X)$ may increase or decrease with X.

Now, if this happens, some observations may have better information than others to estimate the coefficients. Observations with large conditional variance ($Var(e|X)$ is high) will be less precise for estimating Betas (thus should receive less weight in your estimation), on the other hand, those with low variance will be more precise and receive more weight.

Standard OLS assumes all observations are equally important for estimating $\beta s$, so if your model is heteroskedastic, this equal weight assumption will be incorrect, and that will be reflected in the precision of your coefficients. from the analytical point of view you will need to at the very least correct standard errors (other methods for the estimation) if you want to make any time of statistical inference.

HTH

Answer 3

I'm going to answer to test my understanding since the literature on the topic is huge and lots of people on this forum can give better explanations.

I assume we are talking about the ordinary linear regression, not the generalized linear models.

b) Normality - The residual values must be normally distributed ( But, some sources say that the Response variable must be normally distributed. I don't know which one is true)

I also found this confusing. You want the response to be normal conditional to the explanatory variable(s) and this is equivalent to the residuals being normal.

For example, you want to model adult human height as a function of sex. This is considered a well-behaved trait satisfying the linear model assumptions. However, adult height as a whole has a slightly bimodal distribution (one peak for males, one for females) but within males and within females ( i.e. conditional to the explanatory variable sex) the distribution is normal and the assumption satisfied.

c) Independence - The residuals must be independent of each other

Each observation is supposed to give information about the mean of the (normal) distribution it comes from. If observations are correlated, then, intuitively, you are not getting useful information.

Returning to the human height example, if you measure the same male person multiple times you are not getting independent measures for the mean of males group. Consequently, your estimate will be biased by the height of that person. (There are variations of the linear model that account for such repeated measures).

d) Homoscedasticity - The residuals have constant variance for any value of X.

If the variance changes with the mean of X, than the fit will be dominated by observation will large X. For example, you model the weight of different animal species that vary a lot in size, say cats and elephants. The variance within elephants is huge compared to cats' even if the elephants are very similar in size relative to each other. This means that the elephant datapoint have a large influence on the slope of the regression line.

Answer 4

I will here try to answer the "why":

You have decided to fit a linear model to your observations using OLS method, and with only one dependent variable you now have the slope, $\hat{\beta}$, for that variable. Pay attention to the "hat", which means that $\hat{\beta}$ is just the sample estimate of the underlying (unknown) population slope, $\beta$. From $\hat{\beta}$ you have quantified the linear relationship between the indpendent and the dependent variable in your sample, but you are probably interested in the linear relationship in the underlying population.

(1) If the above conditions are met, you suddenly have knowledge about how $\hat{\beta}$ relates to $\beta$, and you can quantify the uncertainty, usually done through a confidence interval (CI). A standard textbook will tell you how. If the above conditions are not met, the confidence interval calculation is not valid, and hence, your conclusions about the population is not valid.

In summary, for a CI to be valid, you need to make sure that the conditions for that specific CI calculation are met. If the conditions are not met, there might be another CI calculation that fits the conditions that your observations actually meet, but maybe not.

(2) There are other desirable properties of $\hat{\beta}$ that depends on the method of estimation and whether the various conditions are met. For example, $\hat{\beta}$ is an unbiased estimator of $\beta$, meaning that the expected value of $\hat{\beta}$ is $\beta$. It is also consistent and efficient, both desirable properties.

To sum up: If none of the conditions are met, you only have a $\hat{\beta}$ but you don't know anything about its uncertainty and how it relates to $\beta$. If all the conditions are met, you know that $\hat{\beta}$ is an unbiased, consistent and efficient estimator for $\beta$, and you can quantify the uncertainty by calculating a CI.