# Why the OLS underestimates the variances of the coefficients

CONSEQUENCES OF HETEROSCEDASTICITY

$$\textbf{1}$$. The presence of heteroscedasticity does not make the OLS estimates of coefficients biased, but it causes the variances of OLS estimates to increase.

$$\textbf{2}$$. The presence of heteroscedasticity causes the OLS to $${\color{Red} {\text{underestimate}}}$$ the variances of the coefficients.

I don't understand why the OLS to underestimate the variances of the coefficients in $$\textbf{2}$$.

The following is my thoughts :

Let $$\mathbf{X}$$ has full column rank， $$\text{the homoscedasticity model} (1):\begin{cases} \mathbf{y}=\mathbf{X} \boldsymbol{\beta}+\varepsilon \\ E(\varepsilon)=\mathbf{0}, \operatorname{Var}(\varepsilon)=\sigma^{2} \mathbf{I} \end{cases};$$

$$\text{the heteroscedasticity model} (2):\begin{cases} \mathbf{y}=\mathbf{X} \boldsymbol{\beta}+\varepsilon \\ E(\varepsilon)=\mathbf{0}, \operatorname{Var}(\varepsilon)=\sigma^{2} \mathbf{V} \end{cases},\text{where} \mathbf{V} \text{ is diagonal but with unequal diagonal elements.}$$

$$\\$$ When $$\text{the heteroscedasticity model} (2)$$ is ture, then the weighted least-squares estimator of $$\hat{\boldsymbol{\beta}}_{WLS}$$ is an unbiased estimator of $$\boldsymbol{\beta}$$ and $$\operatorname{Var}(\hat{\boldsymbol{\beta}}_{WLS})=\sigma^{2}\left(\mathbf{X}^{\prime} \mathbf{V}^{-1} \mathbf{X}\right)^{-1}.$$

$$\\$$ The ordinary least-squares estimator $$\hat{\boldsymbol{\beta}}_{OLS}=\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{y}$$ is no longer appropriate in model $$(2)$$.If the ordinary least squares is used in this case, the resulting estimator $$\hat{\boldsymbol{\beta}}_{OLS}=\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{y}$$ is still unbiased. However, the ordinary least-squares estimator is no longer a minimum variance estimator. That is, the covariance matrix of the ordinary least-squares estimator is $$\operatorname{Var}(\hat{\boldsymbol{\beta}}_{OLS})=\sigma^{2}\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{V} \mathbf{X}\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1}$$ and the covariance matrix of the weighted least-squares estimator $$\sigma^{2}\left(\mathbf{X}^{\prime} \mathbf{V}^{-1} \mathbf{X}\right)^{-1}$$ gives smaller variances for the regression coefficients. So the OLS should overestimates the variances of the coefficients.

• Both these statements are puzzling because they are not generally true and because their implied reference is unclear: "increase" and "underestimate" compared to what, exactly? I therefore suspect there is some implied context where some particular form of heteroscedasticity has been stipulated. Could you supply that context or tell us where these assertions originate?
– whuber
May 11 at 14:36
• Hi @Elisa. When errors are homoskedastic: the error is drawn with equal probability across the sample. When errors are heteroskedastic: errors are a function of values taken by the covariate (or often a function of time). It follows that an assumed constant OLS $\beta$ is estimated with greater uncertainty. Hope this helps. Someone will most likely cast it in more technical terms/give a superior answer. May 12 at 13:08
• @EB3112 Unfortunately, that doesn't follow. Indeed, there are cases where $\beta$ will be estimated with far less uncertainty with weighted least squares. One simple example would be where $x$ can take on the values $1/2$ and $1$ and the conditional response $y$ is normal with mean $x\beta$ and variance $\sigma^2(1-x).$ With a single observation at $x=1$ we can estimate $\beta$ with no uncertainty at all, whereas the OLS estimates will be uncertain no matter how large the sample size.
– whuber
May 12 at 13:58
• Hi @whuber. I shall stay out of any prolonged debates on this platform. I feel my point is sound in terms of an introductory explanation of the problem arising from heteroskedasticity. However, I accept/recognise you can provide a more sophisticated account than I can, and that there are exceptions to my point made above. That said, due to the nature of the question (possibly first year stats) - I don't think they're particularly relevant. May 12 at 14:17
• Hi @whuber. Falsehoods and simplifications are different. On your logic, children aged 5 should begin to learn set theory before counting. But anyway, again, I accept all your points. You know more about stats than I do and are better positioned to write an answer than I am (hence why I wrote a comment). However, it is time for me to log off for now. Best May 12 at 14:59

We need to note that $$\sigma^{2}$$ is an unknown positive constant and $$\mathbf{V}=diag(\lambda_{1},\cdots,\lambda_{n})$$($$\lambda_{i}>0$$ are not all the same; $$\sigma^{2}=0$$ rarely occurs in real-world data).

Assuming Model$$(2)$$ is true, if we misuse the formula$$\operatorname{Var}(\hat{\boldsymbol{\beta}}_{OLS})=\sigma^{2}(\mathbf{X}^{\prime}\mathbf{X})^{-1}$$ which only holds in Model $$(1)$$ to calculate the variances of $$\hat{\boldsymbol{\beta}}_{OLS}=\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{y}$$ in Model $$(2)$$, then we will inevitably get the false result: $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (1)}\right)=\sigma^{2}(\mathbf{X}^{\prime}\mathbf{X})^{-1}$$since the ture value should be $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (2)}\right)=\sigma^{2}\mathbf{(X^{\prime}X)^{-1}} \mathbf{X^{\prime}VX} \mathbf{(X^{\prime}X)^{-1}}.$$

Under the assumption that all eigenvalues $$\lambda_{i}$$ of $$\mathbf{V}$$ are not less than 1,then $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (1)}\right)\leq\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (2)}\right).$$

In fact, It's very easy to explain $$\mathbf{X}^{\prime} \mathbf{(V-I_{n})} \mathbf{X}\ge \mathbf{0}$$ under such assumption. Note that both $$\mathbf{X}^{\prime} \mathbf{V} \mathbf{X}$$ and $$\mathbf{X}^{\prime} \mathbf{X}$$ are positive definite matrices, since $$\mathbf{X}$$ is full column rank.

Actually,whether the true variance of the estimator $$\hat{\boldsymbol{\beta}}_{OLS}$$ for $$\boldsymbol{\beta}$$ in Model (2) is overestimated or underestimated by the ture variance of the estimator $$\hat{\boldsymbol{\beta}}_{OLS}$$ for $$\boldsymbol{\beta}$$ in Model (1) depends on the values of $$\lambda_{i}$$.

The OLS assumes there is no heteroscedasticity, i.e. $$V=I$$, thus $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{O L S}\right)$$ is not the one you mentioned, instead, $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{O L S}\right)=\sigma^2\left(\boldsymbol{X}^T \boldsymbol{X}\right)^{-1}$$. However if the assumption is incorrect, i.e. the heteroscedasticity exists, $$V\neq I$$, this OLS formula will underestimate the variances of the coefficients.

• Welcome to CV, Laz. You repeat the claim -- but what does it mean and why is it correct? Indeed, the formula you supply does not estimate the coefficient variances, because it applies only when $\sigma^2$ is known. And how would you compare the "$\sigma^2$" that appears in the weighted least squares formula (with $\sigma^2 V$ the variance matrix) to the "$\sigma^2$" that appears in the OLS formula (with $\sigma^2$ the common error variance)? They can't mean the same thing except when $V$ is the identity matrix.
– whuber
May 11 at 19:56

Couple of notes:

• Weighted least squares uses a weight matrix $$W$$ and the variance of the parameters is $$\sigma^2 (X^TWX^T)^{-1}$$.
• No particular choice of $$W$$ will bias $$\beta$$, and so classes of these estimators can be compared in terms of their variance.
• Choosing $$W=V^{-1}$$ is a natural choice and, in fact, is the best linear unbiased estimator according to the Gauss-Markov theorem. As such, it's true that, if the variance matrix is anything other than $$\sigma^2 I$$, then $$\sigma^2 (X^TV^{-1}X)^{-1} = \sigma^{*2}(X^TX) + \text{something positive}$$ . Note I have to put a $$^*$$ on the second variance because it's a different parameter.
• The actual variance $$V$$ of a heteroscedastic model is rarely known. For correlated data analysis, the usual process of generalized least squares (GLS) has to do with estimating $$V$$ with $$\hat{V}$$ using the EM algorithm and some assumptions about the covariance structure. You can extend this problem to consider estimating a heteroscedastic variance matrix (with 0 off diagonal entries) using residual analyses, splines, or the like. Like the Fisher Behren's problem, there's no guarantee that this estimator is optimal, and there are cases with the OLS parameter variance is less than (or greater than) any given GLS procedure.

Simple example:

Imagine we estimate just the mean of a sample and we have individuals with the same mean but different variations

$$X_i\sim N(\mu,\sigma_i)$$

The OLS estimate of the mean will be

$$\hat\mu = \frac{\sum_{i=1}^{n} X_i}{n}$$

and has a sampling variation of

$$\text{var}(\hat\mu) = \frac{\sum_{i=1}^{n} \sigma_i^2}{n^2}$$

The estimate of the sampling variation will be based on the sum of squared residuals $$s^2$$ whose mean will be close to...

... (I am getting slightly confused how to deal with this easily)...

... let's try to simulate it

set.seed(1)

sim = function(n=10) {
### create sample
X = c(rnorm(n, mean = 0, sd = 1),
rnorm(n, mean = 0, sd = 0.01))
### linear model
mod = lm(X~1)
### estimate standard error
sqrt(sum(mod\$residuals^2)/(2*n-1))/sqrt(2*n)
}

n  = 10
real = sqrt(n*1^2+n*0.01^2)/(2*n)

s = replicate(10^5,sim())
hist(s, breaks = seq(0,0.35,0.01), xlim = c(0,0.35))

lines(real*c(1,1),c(0,10000), lwd = 2, lty = 2)

mean(s) ## 0.1540672
real    ## 0.1581218 So if we perform OLS a hundred thousand times on simulated samples with size twenty among which ten values of $$\sigma_i = 1$$ and a ten values of $$\sigma_i = 0.01$$, then we get no clear sign of overestimating the sample variance.

Possibly the point 2 is relating to the idea that the sample estimate is not efficient and that some form of generalized least squares will perform better.