23
$\begingroup$

Wikipedia and the R sandwich package vignette give good information about the assumptions supporting OLS coefficient standard errors and the mathematical background of the sandwich estimators. I'm still not clear how the problem of residuals heteroscedasticity is addressed though, probably because I don't fully understand the standard OLS coefficients variance estimation in the first place.

What is the intuition behind the sandwich estimator?

$\endgroup$
3
  • 5
    $\begingroup$ You need to learn more about $M$-estimation (or extremum estimation, as it is sometimes called in econometrics). The sandwich estimator for regression is just a special case of a very general delta-method formula, and if you understand the latter, you won't have any issues with the former. There is no intuition in that the sandwich estimator does not try to model heteroskedasticity or do anything specific about it; it's just a different variance estimator that works under a more general set of assumptions than the standard OLS estimator. $\endgroup$ – StasK Feb 25 '13 at 13:12
  • $\begingroup$ @StasK Thanks! Do you happen to know any particular good resource on M-estimation and delta-method formulas? $\endgroup$ – Robert Kubrick Feb 25 '13 at 14:17
  • $\begingroup$ @Robert Huber's monograph "Robust Statistics" is worth a look. $\endgroup$ – Momo Mar 11 '13 at 21:21
21
$\begingroup$

For OLS, you can imagine that you're using the estimated variance of the residuals (under the assumption of independence and homoscedasticity) as an estimate for the conditional variance of the $Y_i$s. In the sandwich based estimator, you're using the observed squared residuals as a plug-in estimate of the same variance which can vary between observations.

\begin{equation} \mbox{var}\left(\hat{\beta}\right) = \left(X^TX\right)^{-1}\left(X^T\mbox{diag}\left(\mbox{var}\left(Y|X\right)\right)X\right)\left(X^TX\right)^{-1} \end{equation}

In the ordinary least squares standard error estimate for the regression coefficient estimate, the conditional variance of the outcome is treated as constant and independent, so that it can be estimated consistently.

\begin{equation} \widehat{\mbox{var}}_{OLS}\left(\hat{\beta}\right) = \left(X^TX\right)^{-1}\left(r^2X^TX\right)\left(X^TX\right)^{-1} \end{equation}

For the sandwich, we eschew consistent estimation of the conditional variance and instead use a plug-in estimate of the variance of each component using the squared residual

\begin{equation} \widehat{\mbox{var}}_{RSE}\left(\hat{\beta}\right) = \left(X^TX\right)^{-1}\left(X^T\mbox{diag}\left(r_i^2\right)X\right)\left(X^TX\right)^{-1} \end{equation}

By using the plug-in variance estimate, we get consistent estimates of the variance of $\hat{\beta}$ by the Lyapunov Central Limit Theorem.

Intuitively, these observed squared residuals will mop up any unexplained error due to heteroscedasticity that would have otherwise been unexpected under the assumption of constant variance.

$\endgroup$
5
  • $\begingroup$ It's your last paragraph that I have a hard time to grasp. Can you illustrate? $\endgroup$ – Robert Kubrick Feb 25 '13 at 13:00
  • $\begingroup$ It's not SE in your formulae, AdamO, it's SE^2... in whatever matrix way you are going to mean that. $\endgroup$ – StasK Feb 25 '13 at 13:14
  • $\begingroup$ @StasK Good point. Maybe a variance-hat is better. I was confusing multivariate and univariate terminology. $\endgroup$ – AdamO Feb 25 '13 at 16:53
  • 2
    $\begingroup$ @RobertKubrick In the last paragraph, I'm pointing out that the key difference in estimators is how we represent the conditional variance term $\mbox{var}(Y|X)$. In the linear regression model, we consistently estimate the residuals, but with the sandwich, we just use a plug-in estimate of the conditional variance for the $i$-th term using the squared residuals. In the presence of heteroscedasticity, points with relatively large squared residuals have a corresponding large estimated variance and this reduces their influence on the standard error estimates. $\endgroup$ – AdamO Feb 25 '13 at 16:56
  • $\begingroup$ Edit: I said that OLS var estimates involve "consistent estimates of residuals", when I meant to say "consistent estimate of the variance of the residuals". $\endgroup$ – AdamO Nov 18 '13 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.