# Why Do Residuals Need To Be Homoscedastic (Equal Variance)?

I am MBA student and am taking a course on "Research Methods" - in this class, we are learning how to perform basic statistical analysis such as Hypothesis Testing and Regression Models.

Our prof is showing us how to use SAS software and fit a linear regression model to some data. For example, we are predicting how much money a company will earn based on the number of sales that they had.

Prior to finishing the regression analysis, we apparently have to check to make sure that certain assumptions are met. Our prof showed us this page over here which contains these assumptions: https://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/R/R5_Correlation-Regression/R5_Correlation-Regression4.html .

I can indirectly understand why these assumptions are necessary, but our prof isn't really capable to explaining the consequences of fitting regression models when these assumptions are not satisfied. For example, our prof simulated a fake data set in which some of the assumptions are not met. Then he fit a regression model to this data and showed how poor the performance was in hopes that this would convince us about the importance of these assumptions. While I do accept his authority on this matter, I can't help but wonder if this example that he showed us was a "one off" and not necessarily the general case (e.g. some datasets might not meet these assumptions yet still produce well-performing regression models).

For instance, suppose we considered the example of the "heteroscedastic" assumption : the residuals of the linear regression model MUST have equal variance (homoscedastic) - what happens if this is NOT the case? Is it possible to somehow prove that when the "heteroscedastic" assumption is not satisfied, this will statistically provide worse regression model in similar circumstances had the residuals satisfied the "heteroscedastic" assumption? Maybe the "heteroscedastic" assumption results in the regression coefficients having wider confidence intervals?

• The issues of normality and homoscedasticity are distinct. For instance, you can have normality without it constant variance, and you can have non-normality with constant variance.
– Dave
Sep 10, 2022 at 2:09
• Lack of hetroscedastic can also indicate a poorly chosen model or a model missing key parameter(s). The hetroscedastic assumption is to ensure your prediction is equally accurate across the range of the model. If the variance changes based on the value of the independent variable then the prediction will go from good to bad without any warning to the user. Sep 10, 2022 at 3:39
• +1 These are thoughtful questions. Your conjecture at the end is generally correct, but a specific answer depends on just what you mean by "heteroscedastic." Some people understand this very broadly as any kind of variation in the spread and shape of the conditional response distributions; but in most regression settings it is limited to variation in the spread alone. In this narrower case, OLS estimates are still unbiased (caeteris paribus) but estimates of uncertainty, such as confidence limits and prediction limits, can be poor and will be improved using weighted least squares methods.
– whuber
Sep 10, 2022 at 15:46
– Dave
Nov 6, 2022 at 4:55
• The more general topic of testing model assumption seems relevant; see for example 1 and 2. Nov 6, 2022 at 17:53

The bounty asks for citations. Some relevant citations appear in comments to Literature on robustness of regression assumptions. I reproduce them here:

Whuber:

Draper & Smith (Applied Regression Analysis, 2nd Ed.) develop the regression equations at the beginning of section 2.6, then discuss what can be done in a subsection "Without Distributional Assumptions," and only then discuss what can further be done (mainly with the F tests) in a subsection "With Distributional Assumptions." Ultimately, "robustness" is going to be relative to the conclusions you are trying to draw: some of them will be largely insensitive to homoscedasticity but others might be more sensitive.

Ben Bolker:

• Coombs, William T., James Algina, and Debra Olson Oltman. "Univariate and multivariate omnibus hypothesis tests selected to control type I error rates when population variances are not necessarily equal." Review of Educational Research 66.2 (1996): 137-179 ;

• Tomarken, Andrew J., and Ronald C. Serlin. "Comparison of ANOVA alternatives under variance heterogeneity and specific noncentrality structures." Psychological Bulletin 99.1 (1986): 90.

Incidentally, I found this thread by using the Search feature. This is exact string that I used homoscedastic [references] answers:1 and it was the first result. More information about how to effectively search the site is here: FAQ: Best Practices for Searching CV.

The assumptions of residual normality and homoscedaticity are used in the theoretical derivation of the Mean Squared Error. This loss is actually a scaled negative logarithmic likelihood for the following regression model:

$$y = f(\theta, x) + \epsilon$$

where $$\epsilon$$ is a zero mean normal random variable, independent of $$x$$.

If we use Mean Squared Error in our optimization task and the residuals turn out to be heteroscedatic, that means that our model was incorrect from the beginning.

If we want to retain residual normality, but capture heteroscedaticity, we can consider a more complex parametric model:

$$y = f(\theta, x) + g(\theta, x)\epsilon$$

where $$\epsilon \sim N(0, 1)$$ is independent of $$x$$.

The loss function to be optimized here will be different:

$$\sum_{i = 1}^N \ln(g(\theta, x_i)) + (\frac{y_i - f(\theta, x_i)}{g(\theta, x_i)})^2$$

I have written quite a bit on model assumptions here and here.

Just to add a general remark to the discussion: Statistical model assumptions are required for proving mathematical theorems regarding the quality of a method. In case model assumptions are violated, the theoretical "guarantees" of the theorems cannot be taken for granted (this includes straightforward things as the correctness of p-values).

Now we have to keep in mind that the mathematical model and its formal assumptions live in the mathematical formal world and not in the real world, in which, as George Box famously wrote, "all models are wrong but some are useful". The same holds for the theorems concerning the methods.

If model assumptions do not hold, theory does not guarantee good behaviour (although there are a few exceptions where theory also exists that characterises behaviour outside the nominal model, the Central Limit Theorem arguably being an example in case of violation of normality). This does not necessarily mean that the behaviour is bad! One idea that is often involved is that if model violations are very mild, they may well be harmless. This is often true but unfortunately not always, and investigating which violations of assumptions cause what kind of trouble and when it is harmless and when rather not is a box of Pandora, things depend strongly on what exactly goes on (for example to what extent heteroscedasticity in a two-sample situation is harmful depends on whether the larger variance occurs in the larger or in the smaller sample). On the positive side, some quite substantial violations of model assumptions do not cause much harm (some very obviously non-normal distributions play very nicely with inference based on the normal assumption).

Regarding heteroscedasticity, this means that we observe different amounts of variation in different places in $$x$$-space, which means that observations in different places give us differently strong information about where the regression line should be. As the LS-regression estimator treats all observations in the same way, it should be clear that better estimators could be defined weighting more informative observations up (but one would need to know some detail about the exact shape of heteroscedasticity to choose them). This means in the first place that the LS-estimator loses optimality, however it may still be OK; to some extent this depends on how much precision is required, and how strong the heteroscedasticity is. Also the LS-estimator will assume for prediction that precision is everywhere the same, whereas you may want to be warned that in some places much less precision for prediction can be achieved. Regarding p-values of associated tests, I'd suspect that some power will be lost, but I don't expect the effect to be strong as long as heteroscedasticity is mild.

Often heteroscedasticity is a hint that the linear model works better with transformed variables; switching for example from $$y$$ to $$\log y$$ or $$\sqrt y$$ (maybe of $$y+c$$) has often more visible effect on heteroscedasticity than on linearity, but may improve the model a lot.

• So the message about "how important is that model assumptions hold?"" is really "how important is that assumptions hold for the particular model I'm using given the actual data I'm analyzing and the goals I have for the analysis?"? There is no "general case". Also, the second link doesn't seem to work. Nov 7, 2022 at 0:57
• @dipetkov Link repaired, thanks! Part of my message is that "how important is that model assumptions hold?" is the wrong question, because they won't ever hold anyway. And yes, unfortunately in practice there is no general case, and one needs to look at the specific application and data set. Nov 7, 2022 at 16:30