Sandwich estimator intuition

Wikipedia and the R sandwich package vignette give good information about the assumptions supporting OLS coefficient standard errors and the mathematical background of the sandwich estimators. I'm still not clear how the problem of residuals heteroscedasticity is addressed though, probably because I don't fully understand the standard OLS coefficients variance estimation in the first place.

What is the intuition behind the sandwich estimator?

• You need to learn more about $M$-estimation (or extremum estimation, as it is sometimes called in econometrics). The sandwich estimator for regression is just a special case of a very general delta-method formula, and if you understand the latter, you won't have any issues with the former. There is no intuition in that the sandwich estimator does not try to model heteroskedasticity or do anything specific about it; it's just a different variance estimator that works under a more general set of assumptions than the standard OLS estimator. Feb 25, 2013 at 13:12
• @StasK Thanks! Do you happen to know any particular good resource on M-estimation and delta-method formulas? Feb 25, 2013 at 14:17
• @Robert Huber's monograph "Robust Statistics" is worth a look.
– Momo
Mar 11, 2013 at 21:21

For OLS, you can imagine that you're using the estimated variance of the residuals (under the assumption of independence and homoscedasticity) as an estimate for the conditional variance of the $Y_i$s. In the sandwich based estimator, you're using the observed squared residuals as a plug-in estimate of the same variance which can vary between observations.

$$\mbox{var}\left(\hat{\beta}\right) = \left(X^TX\right)^{-1}\left(X^T\mbox{diag}\left(\mbox{var}\left(Y|X\right)\right)X\right)\left(X^TX\right)^{-1}$$

In the ordinary least squares standard error estimate for the regression coefficient estimate, the conditional variance of the outcome is treated as constant and independent, so that it can be estimated consistently.

$$\widehat{\mbox{var}}_{OLS}\left(\hat{\beta}\right) = \left(X^TX\right)^{-1}\left(r^2X^TX\right)\left(X^TX\right)^{-1}$$

For the sandwich, we eschew consistent estimation of the conditional variance and instead use a plug-in estimate of the variance of each component using the squared residual

$$\widehat{\mbox{var}}_{RSE}\left(\hat{\beta}\right) = \left(X^TX\right)^{-1}\left(X^T\mbox{diag}\left(r_i^2\right)X\right)\left(X^TX\right)^{-1}$$

By using the plug-in variance estimate, we get consistent estimates of the variance of $\hat{\beta}$ by the Lyapunov Central Limit Theorem.

Intuitively, these observed squared residuals will mop up any unexplained error due to heteroscedasticity that would have otherwise been unexpected under the assumption of constant variance.

• It's your last paragraph that I have a hard time to grasp. Can you illustrate? Feb 25, 2013 at 13:00
• It's not SE in your formulae, AdamO, it's SE^2... in whatever matrix way you are going to mean that. Feb 25, 2013 at 13:14
• @StasK Good point. Maybe a variance-hat is better. I was confusing multivariate and univariate terminology. Feb 25, 2013 at 16:53
• @RobertKubrick In the last paragraph, I'm pointing out that the key difference in estimators is how we represent the conditional variance term $\mbox{var}(Y|X)$. In the linear regression model, we consistently estimate the residuals, but with the sandwich, we just use a plug-in estimate of the conditional variance for the $i$-th term using the squared residuals. In the presence of heteroscedasticity, points with relatively large squared residuals have a corresponding large estimated variance and this reduces their influence on the standard error estimates. Feb 25, 2013 at 16:56
• Edit: I said that OLS var estimates involve "consistent estimates of residuals", when I meant to say "consistent estimate of the variance of the residuals". Nov 18, 2013 at 17:41