Why Do Residuals Need To Be Homoscedastic (Equal Variance)?

Question

I am MBA student and am taking a course on "Research Methods" - in this class, we are learning how to perform basic statistical analysis such as Hypothesis Testing and Regression Models.

Our prof is showing us how to use SAS software and fit a linear regression model to some data. For example, we are predicting how much money a company will earn based on the number of sales that they had.

Prior to finishing the regression analysis, we apparently have to check to make sure that certain assumptions are met. Our prof showed us this page over here which contains these assumptions: https://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/R/R5_Correlation-Regression/R5_Correlation-Regression4.html .

I can indirectly understand why these assumptions are necessary, but our prof isn't really capable to explaining the consequences of fitting regression models when these assumptions are not satisfied. For example, our prof simulated a fake data set in which some of the assumptions are not met. Then he fit a regression model to this data and showed how poor the performance was in hopes that this would convince us about the importance of these assumptions. While I do accept his authority on this matter, I can't help but wonder if this example that he showed us was a "one off" and not necessarily the general case (e.g. some datasets might not meet these assumptions yet still produce well-performing regression models).

For instance, suppose we considered the example of the "heteroscedastic" assumption : the residuals of the linear regression model MUST have equal variance (homoscedastic) - what happens if this is NOT the case? Is it possible to somehow prove that when the "heteroscedastic" assumption is not satisfied, this will statistically provide worse regression model in similar circumstances had the residuals satisfied the "heteroscedastic" assumption? Maybe the "heteroscedastic" assumption results in the regression coefficients having wider confidence intervals?

Answer 1

The bounty asks for citations. Some relevant citations appear in comments to Literature on robustness of regression assumptions. I reproduce them here:

Whuber:

Draper & Smith (Applied Regression Analysis, 2nd Ed.) develop the regression equations at the beginning of section 2.6, then discuss what can be done in a subsection "Without Distributional Assumptions," and only then discuss what can further be done (mainly with the F tests) in a subsection "With Distributional Assumptions." Ultimately, "robustness" is going to be relative to the conclusions you are trying to draw: some of them will be largely insensitive to homoscedasticity but others might be more sensitive.

Ben Bolker:

• Coombs, William T., James Algina, and Debra Olson Oltman. "Univariate and multivariate omnibus hypothesis tests selected to control type I error rates when population variances are not necessarily equal." Review of Educational Research 66.2 (1996): 137-179 ;

• Tomarken, Andrew J., and Ronald C. Serlin. "Comparison of ANOVA alternatives under variance heterogeneity and specific noncentrality structures." Psychological Bulletin 99.1 (1986): 90.

Incidentally, I found this thread by using the Search feature. This is exact string that I used homoscedastic [references] answers:1 and it was the first result. More information about how to effectively search the site is here: FAQ: Best Practices for Searching CV.

Answer 2

The assumptions of residual normality and homoscedaticity are used in the theoretical derivation of the Mean Squared Error. This loss is actually a scaled negative logarithmic likelihood for the following regression model:

$$y = f(\theta, x) + \epsilon$$

where $\epsilon$ is a zero mean normal random variable, independent of $x$.

If we use Mean Squared Error in our optimization task and the residuals turn out to be heteroscedatic, that means that our model was incorrect from the beginning.

If we want to retain residual normality, but capture heteroscedaticity, we can consider a more complex parametric model:

$$y = f(\theta, x) + g(\theta, x)\epsilon$$

where $\epsilon \sim N(0, 1)$ is independent of $x$.

The loss function to be optimized here will be different:

$$\sum_{i = 1}^N \ln(g(\theta, x_i)) + (\frac{y_i - f(\theta, x_i)}{g(\theta, x_i)})^2$$

Answer 3

I have written quite a bit on model assumptions here and here.

Just to add a general remark to the discussion: Statistical model assumptions are required for proving mathematical theorems regarding the quality of a method. In case model assumptions are violated, the theoretical "guarantees" of the theorems cannot be taken for granted (this includes straightforward things as the correctness of p-values).

Now we have to keep in mind that the mathematical model and its formal assumptions live in the mathematical formal world and not in the real world, in which, as George Box famously wrote, "all models are wrong but some are useful". The same holds for the theorems concerning the methods.

If model assumptions do not hold, theory does not guarantee good behaviour (although there are a few exceptions where theory also exists that characterises behaviour outside the nominal model, the Central Limit Theorem arguably being an example in case of violation of normality). This does not necessarily mean that the behaviour is bad! One idea that is often involved is that if model violations are very mild, they may well be harmless. This is often true but unfortunately not always, and investigating which violations of assumptions cause what kind of trouble and when it is harmless and when rather not is a box of Pandora, things depend strongly on what exactly goes on (for example to what extent heteroscedasticity in a two-sample situation is harmful depends on whether the larger variance occurs in the larger or in the smaller sample). On the positive side, some quite substantial violations of model assumptions do not cause much harm (some very obviously non-normal distributions play very nicely with inference based on the normal assumption).

Regarding heteroscedasticity, this means that we observe different amounts of variation in different places in $x$-space, which means that observations in different places give us differently strong information about where the regression line should be. As the LS-regression estimator treats all observations in the same way, it should be clear that better estimators could be defined weighting more informative observations up (but one would need to know some detail about the exact shape of heteroscedasticity to choose them). This means in the first place that the LS-estimator loses optimality, however it may still be OK; to some extent this depends on how much precision is required, and how strong the heteroscedasticity is. Also the LS-estimator will assume for prediction that precision is everywhere the same, whereas you may want to be warned that in some places much less precision for prediction can be achieved. Regarding p-values of associated tests, I'd suspect that some power will be lost, but I don't expect the effect to be strong as long as heteroscedasticity is mild.

Often heteroscedasticity is a hint that the linear model works better with transformed variables; switching for example from $y$ to $\log y$ or $\sqrt y$ (maybe of $y+c$) has often more visible effect on heteroscedasticity than on linearity, but may improve the model a lot.