One of the assumptions of linear regression is that there should be a constant variance in the error terms and that the confidence intervals and hypothesis tests associated with the model rely on this assumption. What exactly happens when the error terms do not have a constant variance?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
5
The consequences of heteroscedasticity are:
The ordinary least squares (OLS) estimator $\hat{\mathbf{b}} = \left(X'X \right)X'\mathbf{y}$ is still consistent but it is no longer efficient.
The estimate $\hat{\mathrm{Var}}\left(\mathbf{b} \right) = \left( X'X\right)^{-1} \hat{\sigma}^2$ where $\hat{\sigma}^2 = \frac{1}{n-k} \mathbf{e'}{\mathbf{e}}$ is not a consistent estimator anymore for the covariance matrix of your estimator $\hat{\mathbf{b}}$. It may be both biased and inconsistent. And in practice, it can substantially underestimate the variance.
Point (1) may not be a major issue; people often use the ordinary OLS estimator anyway. But point (2) must be addressed. What to do?
You need heteroscedasticity-consistent standard errors. The standard approach is to lean on large-sample assumptions, asymptotic results and estimate the variance of $\mathbf{b}$ using:
$$\hat{\mathrm{Var}}\left(\mathbf{b}\right)=\frac{1}{n}\left( \frac{X'X}{n} \right)^{-1} S \left( \frac{X'X}{n} \right)^{-1}$$ where $S$ is estimated as $S = \frac{1}{n-k}\sum_i \left(\mathbf{x}_i e_i\right) \left(\mathbf{x}_i e_i \right)'$.
This gives heteroskedasticity-consistent standard errors. They're also known as Huber-White standard errors, robust standard errors, "sandwich" estimator, etc... Any basic standard statistics package has an option for robust standard errors. Use it!
Some additional comments (update)
If the heteroskedasticity is large enough, the regular OLS estimate can have big practical problems. While still a consistent estimator, you may have small sample problems where your whole estimate is driven by a few, high variance observations. (This is what @seanv507 is alluding to in comments). The OLS estimator is inefficient in that it's giving more weight to high variance observations than optimal. The estimate may be extremely noisy.
A problem with trying to fix the inefficiency is that you probably don't know the covariance matrix for the error terms either, hence using something like GLS can make things even worse if your estimate of the error term covariance matrix is garbage.
Also, the Huber-White standard errors I give above may have big problems in small samples. There is a long literature on this topic. Eg. see Imbens and Kolesar (2016), "Robust Standard Errors in Small Samples: Some Practical Advice."
Direction for further study:
If this is self-study, the next practical thing to consider are clustered standard errors. These correct for arbitrary correlation within clusters.
answered Oct 17, 2016 at 8:46
-
1$\begingroup$ Matthew - I think more practical problems would clarify point (1). eg wouldn't the estimator be 'biased' towards those regions with higher variance? - which would be a bigger problem if those regions were far from the mean causing high leverage. $\endgroup$– seanv507Commented Oct 17, 2016 at 9:05
-
3$\begingroup$ @seanv507 heteroskedasticity doesn't bias the OLS estimate. What I think you're referring to is inefficiency. By weighting high-variance observations and low-variance observations equally, the OLS estimator has higher variance than is theoretically achievable with something like inverse variance weighting. Whether you want to use your estimates of $\sigma^2_i$ in the estimation phase (i.e. for estimating $\mathbf{b}$) depends on how much you believe you know $\sigma^2_i$. $\endgroup$ Commented Oct 17, 2016 at 9:17
-
1$\begingroup$ Matthew, I know its not introducing bias (I apologise [to you and OP] for using the term in quotes :) I couldn't think of the appropriate term ). But I am trying to draw out the practical implications (and suggesting the OP wants to understand those) - when/why point (1) is not a major issue. Wouldn't you agree that the effect is that then $ \mathbb b$ depends more on the high variance region than you might intuitively expect/want .(intuitive straight line fit would be that each region has equal weighting whereas infact OLS will concentrate more on high variance regions). $\endgroup$– seanv507Commented Oct 17, 2016 at 9:47
-
$\begingroup$ @seanv507 feel free to add your own answer! $\endgroup$ Commented Oct 17, 2016 at 10:09
-
$\begingroup$ In place of using heteroskedasticity-robust standard errors (which Ed Leamer in his 2010 paper "Tantalus on the road to Asymptopia" calls White-washing), one could also try to correct the point estimates (together with the variance estimate) for heteroskedasticity by WLS. This might be worth mentioning in your answer. $\endgroup$ Commented Apr 30, 2019 at 13:45
$\begingroup$
$\endgroup$
Well the short answer is basically your model is wrong i.e.
- In order for the ordinary least squares to be the Best Linear Unbiased Estimator the constant variance of the error terms is assumed.
- The Gauss-Markov assumptions - if fulfilled - guarantee you that the least squares estimator for the coefficents $\beta$ is unbiased and has a min variance amongst all unbiased linear estimators.
So in case of heteroscedasticity problems with estimating the variance-covariance matrix happen, which lead to wrong standard errors of the coefficients, which in turn leads to wrong t-statistics and p-values. Briefly put, if your error terms do not have constant variance then ordinary least squares is not the most efficient way for estimation. Have a look at this related question.
$\begingroup$
$\endgroup$
"Heteroscedasticity" makes it difficult to estimate the true standard deviation of the forecast errors. This can lead to confidence intervals that are too wide or too narrow (in particular they will be too narrow for out-of-sample predictions, if the variance of the errors is increasing over time).
Also, the regression model may focus too heavily on a subset of data.
Good reference: Testing assumptions of linear regression
