# Why the OLS underestimates the variances of the coefficients

CONSEQUENCES OF HETEROSCEDASTICITY

$$\textbf{1}$$. The presence of heteroscedasticity does not make the OLS estimates of coefficients biased, but it causes the variances of OLS estimates to increase.

$$\textbf{2}$$. The presence of heteroscedasticity causes the OLS to $${\color{Red} {\text{underestimate}}}$$ the variances of the coefficients.

I don't understand why the OLS to underestimate the variances of the coefficients in $$\textbf{2}$$.

The following is my thoughts :

Let $$\mathbf{X}$$ has full column rank， $$\text{the homoscedasticity model} (1):\begin{cases} \mathbf{y}=\mathbf{X} \boldsymbol{\beta}+\varepsilon \\ E(\varepsilon)=\mathbf{0}, \operatorname{Var}(\varepsilon)=\sigma^{2} \mathbf{I} \end{cases};$$

$$\text{the heteroscedasticity model} (2):\begin{cases} \mathbf{y}=\mathbf{X} \boldsymbol{\beta}+\varepsilon \\ E(\varepsilon)=\mathbf{0}, \operatorname{Var}(\varepsilon)=\sigma^{2} \mathbf{V} \end{cases},\text{where} \mathbf{V} \text{ is diagonal but with unequal diagonal elements.}$$

$$\\$$ When $$\text{the heteroscedasticity model} (2)$$ is ture, then the weighted least-squares estimator of $$\hat{\boldsymbol{\beta}}_{WLS}$$ is an unbiased estimator of $$\boldsymbol{\beta}$$ and $$\operatorname{Var}(\hat{\boldsymbol{\beta}}_{WLS})=\sigma^{2}\left(\mathbf{X}^{\prime} \mathbf{V}^{-1} \mathbf{X}\right)^{-1}.$$

$$\\$$ The ordinary least-squares estimator $$\hat{\boldsymbol{\beta}}_{OLS}=\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{y}$$ is no longer appropriate in model $$(2)$$.If the ordinary least squares is used in this case, the resulting estimator $$\hat{\boldsymbol{\beta}}_{OLS}=\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{y}$$ is still unbiased. However, the ordinary least-squares estimator is no longer a minimum variance estimator. That is, the covariance matrix of the ordinary least-squares estimator is $$\operatorname{Var}(\hat{\boldsymbol{\beta}}_{OLS})=\sigma^{2}\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{V} \mathbf{X}\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1}$$ and the covariance matrix of the weighted least-squares estimator $$\sigma^{2}\left(\mathbf{X}^{\prime} \mathbf{V}^{-1} \mathbf{X}\right)^{-1}$$ gives smaller variances for the regression coefficients. So the OLS should overestimates the variances of the coefficients.

• Both these statements are puzzling because they are not generally true and because their implied reference is unclear: "increase" and "underestimate" compared to what, exactly? I therefore suspect there is some implied context where some particular form of heteroscedasticity has been stipulated. Could you supply that context or tell us where these assertions originate?
– whuber
Commented May 11, 2023 at 14:36
• Hi @Elisa. When errors are homoskedastic: the error is drawn with equal probability across the sample. When errors are heteroskedastic: errors are a function of values taken by the covariate (or often a function of time). It follows that an assumed constant OLS $\beta$ is estimated with greater uncertainty. Hope this helps. Someone will most likely cast it in more technical terms/give a superior answer. Commented May 12, 2023 at 13:08
• @EB3112 Unfortunately, that doesn't follow. Indeed, there are cases where $\beta$ will be estimated with far less uncertainty with weighted least squares. One simple example would be where $x$ can take on the values $1/2$ and $1$ and the conditional response $y$ is normal with mean $x\beta$ and variance $\sigma^2(1-x).$ With a single observation at $x=1$ we can estimate $\beta$ with no uncertainty at all, whereas the OLS estimates will be uncertain no matter how large the sample size.
– whuber
Commented May 12, 2023 at 13:58
• @EB3112 You can't get started learning when you begin with falsehoods.
– whuber
Commented May 12, 2023 at 14:49
• Hi @whuber. Falsehoods and simplifications are different. On your logic, children aged 5 should begin to learn set theory before counting. But anyway, again, I accept all your points. You know more about stats than I do and are better positioned to write an answer than I am (hence why I wrote a comment). However, it is time for me to log off for now. Best Commented May 12, 2023 at 14:59

We need to note that $$\sigma^{2}$$ is an unknown positive constant and $$\mathbf{V}=diag(\lambda_{1},\cdots,\lambda_{n})$$($$\lambda_{i}>0$$ are not all the same; $$\sigma^{2}=0$$ rarely occurs in real-world data).

Assuming Model$$(2)$$ is true, if we misuse the formula$$\operatorname{Var}(\hat{\boldsymbol{\beta}}_{OLS})=\sigma^{2}(\mathbf{X}^{\prime}\mathbf{X})^{-1}$$ which only holds in Model $$(1)$$ to calculate the variances of $$\hat{\boldsymbol{\beta}}_{OLS}=\left(\mathbf{X}^{\prime} \mathbf{X}\right)^{-1} \mathbf{X}^{\prime} \mathbf{y}$$ in Model $$(2)$$, then we will inevitably get the false result: $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (1)}\right)=\sigma^{2}(\mathbf{X}^{\prime}\mathbf{X})^{-1}$$since the ture value should be $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (2)}\right)=\sigma^{2}\mathbf{(X^{\prime}X)^{-1}} \mathbf{X^{\prime}VX} \mathbf{(X^{\prime}X)^{-1}}.$$

Under the assumption that all eigenvalues $$\lambda_{i}$$ of $$\mathbf{V}$$ are not less than 1,then $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (1)}\right)\leq\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{OLS}\big|\text{ Model (2)}\right).$$

In fact, It's very easy to explain $$\mathbf{X}^{\prime} \mathbf{(V-I_{n})} \mathbf{X}\ge \mathbf{0}$$ under such assumption. Note that both $$\mathbf{X}^{\prime} \mathbf{V} \mathbf{X}$$ and $$\mathbf{X}^{\prime} \mathbf{X}$$ are positive definite matrices, since $$\mathbf{X}$$ is full column rank.

Actually,whether the true variance of the estimator $$\hat{\boldsymbol{\beta}}_{OLS}$$ for $$\boldsymbol{\beta}$$ in Model (2) is overestimated or underestimated by the ture variance of the estimator $$\hat{\boldsymbol{\beta}}_{OLS}$$ for $$\boldsymbol{\beta}$$ in Model (1) depends on the values of $$\lambda_{i}$$.

The OLS assumes there is no heteroscedasticity, i.e. $$V=I$$, thus $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{O L S}\right)$$ is not the one you mentioned, instead, $$\operatorname{Var}\left(\hat{\boldsymbol{\beta}}_{O L S}\right)=\sigma^2\left(\boldsymbol{X}^T \boldsymbol{X}\right)^{-1}$$. However if the assumption is incorrect, i.e. the heteroscedasticity exists, $$V\neq I$$, this OLS formula will underestimate the variances of the coefficients.

• Welcome to CV, Laz. You repeat the claim -- but what does it mean and why is it correct? Indeed, the formula you supply does not estimate the coefficient variances, because it applies only when $\sigma^2$ is known. And how would you compare the "$\sigma^2$" that appears in the weighted least squares formula (with $\sigma^2 V$ the variance matrix) to the "$\sigma^2$" that appears in the OLS formula (with $\sigma^2$ the common error variance)? They can't mean the same thing except when $V$ is the identity matrix.
– whuber
Commented May 11, 2023 at 19:56
• weighted sum of products in Linear Algebra Commented Mar 30 at 13:13

Couple of notes:

• Weighted least squares uses a weight matrix $$W$$ and the variance of the parameters is $$\sigma^2 (X^TWX^T)^{-1}$$.
• No particular choice of $$W$$ will bias $$\beta$$, and so classes of these estimators can be compared in terms of their variance.
• Choosing $$W=V^{-1}$$ is a natural choice and, in fact, is the best linear unbiased estimator according to the Gauss-Markov theorem. As such, it's true that, if the variance matrix is anything other than $$\sigma^2 I$$, then $$\sigma^2 (X^TV^{-1}X)^{-1} = \sigma^{*2}(X^TX) + \text{something positive}$$ . Note I have to put a $$^*$$ on the second variance because it's a different parameter.
• The actual variance $$V$$ of a heteroscedastic model is rarely known. For correlated data analysis, the usual process of generalized least squares (GLS) has to do with estimating $$V$$ with $$\hat{V}$$ using the EM algorithm and some assumptions about the covariance structure. You can extend this problem to consider estimating a heteroscedastic variance matrix (with 0 off diagonal entries) using residual analyses, splines, or the like. Like the Fisher Behren's problem, there's no guarantee that this estimator is optimal, and there are cases with the OLS parameter variance is less than (or greater than) any given GLS procedure.

OLS will split up the observations into two orthogonal subspaces. See: Why are the residuals in $\mathbb{R}^{n-p}$?

The homoscedasticity assumption of OLS is equivalent to the assumption that the observations are spherically symmetric distributed and that the partition of sum of squares of the projections onto the two spaces is related to the dimensions of the two spaces.

This assumption is false when the errors are heteroscedastic. Depending on the orientation of the error distribution relative to the space of the model, the relationship of the two variances can be smaller or larger. So it depends and the estimates of the variances of the coëfficiënts can both be overestimated as well as underestimated.

Below is a simple example that illustrates the different cases. The situation is for a model

$$y_i \sim N(\mu x_i, \sigma_i^2)$$

and we have $$x_1,x_2 = 1,2$$ and we vary $$\sigma_i$$.

The images show each 100 simulations of samples of size two, $$y_1,y_2$$ and their orthogonal projection onto the line $$\hat{y}_2 = 2 \hat{y}_1$$, which is the OLS estimate.

• In the first image the errors are homoscedastic, and the OLS will correctly assume that the distribution of the residual has the same variance as the distribution of the estimate.
• In the second and third images the errors are heteroscedastic.
• In the second image, when $$\sigma_2 < \sigma_1$$ the variance of the residuals is relatively larger than the variance of the estimate. OLS will overestimate the variance of the coëfficiënts.
• In the third image, when $$\sigma_2 > \sigma_1$$ the variance of the residuals is relatively smaller than the variance of the estimate. OLS will underestimate the variance of the coëfficiënts.

• "sum of squared residuals s2 whose mean will be close to..." ZERO for Normalized Data (aka demeaned & divided by std)... to get init_data after fit do reverse operations multiply solution ([email protected]) by ystd & add to ymean Commented Mar 30 at 13:09
• Resampling & Simulation Commented Mar 30 at 13:18
• @JeeyCi I do not understand what your comments were supposed to mean (the sum of squared residuals is not zero, unless all residuals are zero, because you are adding non-negative numbers), but your comment made me see this old post, and I have changed it drastically. Commented Mar 30 at 15:28
• "your comment made me see this old post, and I have changed it drastically." - exactly, your dots in your previous statement were confusing - like you didn't know what to do with residuals. & mean.. I agree, concerning Sum of squares (if my statement occured confusing for you), but concerning mean of residuals - it equals zero in OLS & residuals being normally distributed, otherwise scaling or intercept needed or if still any pattern exists in residuals - change the model to incorporate it (this pattern) as well Commented Apr 5 at 6:40
• Still extrapolation of the model can be questionable under certain conditions. Commented Apr 5 at 6:48