How does the assumption of constant variance affect the inferential tools

Question

I'm learning applied linear regression right now and of course one important topic is constant variance of errors. In Sheather's textbook, it's said that

Ignoring nonconstant variance when it exists invalidates all inferential tools (i.e. p-values, confidence intervals, prediction intervals etc.).

But he doesn't explain the reason. Can someone explain it from a statistical perspective to show how the violation of constant variance assumption will invalidate the inferential tools?

To go more straightforward: Why do we need the assumption of constant variance? I found some relevant answers on Cross Validated like here, but they seem not able to make me clear.

Thanks!

Answer 1

The simple regression linear model looks like $y = X\beta + \epsilon$ and among constraints we have independent white noise $\epsilon_i \sim N(0, \sigma^2)$. Fitting with least mean squares gives the know normal equations in the form:

$\hat{\beta} = (X^TX)^{-1}X^Ty$

To optimize with least mean square one does not use any statistical reasoning. However if you ask questions about the statistical properties of your sample statistic $\hat{\beta}$ you start too need those assumptions.

For example if you ask about statistical properties of your sample statistics this is something else. For example if we ask if the $\hat{\beta}$ is biased you might go with:

$E\hat{\beta} = E[(X^TX)^{-1}X^Ty] = E[(X^TX)^{-1}X^T(X\beta+\epsilon)] = E\beta + E[(X^TX)^{-1}X^T\epsilon]$

If we consider X as fixed constant we have that $E\hat{\beta} = E\beta + (X^TX)^{-1}X^TE[\epsilon] = \beta$ only if $\epsilon$ have independent observations.

If we go for variance of $\hat{\beta}$ we see that has a fixed component $\beta$ and a random noise $(X^TX)^{-1}X^T\epsilon$. So to find the variance of our estimator we look for

$Var(\hat{\beta}) = Var[\beta + (X^TX)^{-1}X^T\epsilon]$

Here the $\beta$ in variance have no effect, we can remove it. See properties of variance here: basic properties of variance on wikipedia. We also use the assumption that $X$ is given so it behaves like a constant term which factors out squared. We have then

$Var(\hat{\beta}) = ((X^TX)^{-1}X^T)^2 Var(\epsilon) = (X^TX)^{-1}Var(\epsilon)$

Now the answer to your question is, if we assume that variance is the same for all noise (additionally to independence) then we can say that

$Var(\beta) = \sigma^2 (X^TX)^{-1}$

And by estimating $\sigma$ you have all the good stuff about coefficients like standard errors, t values, p values, confidence intervals, hypothesis tests and so on. If the variance is not assumed to by heteroskedastic then you have to control it somehow, but things gets more complicated.

Even if the $X$ is not assumed as fixed variable, if is considered random variable a similar assumption on independence like $E[X\epsilon] = 0$ is enough together with heteroskedasticity to have the same results.

Hope that helps.

Answer 2

When we do linear regression, it's under the assumption of Homoscedasticity, which means that the variance of the errors is constant. Knowing that $\sigma^2$ is constant, and not a random variable, it's possible to perform inference using the ordinary least squares estimates.

If the assumption it's not valid, meaning thta you have a heterocedastic model, the estimation by ordinary least squares is no longer valid.

I actually think the answer you linked here is quite clear. Can you specify what you didn't understood?