Variance estimation for regression coefficients with complex survey data

Question

I would like to explain to one of my professors why the regular methods for variance estimation are biased for complex survey data, but I am having some trouble articulating my point.

With weighted least squares in general, the variance for the regression coefficient is $var(\hat{\beta})=\sigma^2(X'WX)^{-1}$

This assumes each observation is independent, and doesn't take into account the probability of selection.

In practice, variances would be calculated using either Taylor Series linearizaton so $var(\hat{\beta})=(X'WX)^{-1}G(X'WX)^{-1}$ where $G$ is essentially the sum of the covariance matrices within stratas, or using resampling via bootstrap or jackknife.

What I am having trouble demonstrating is why exactly $\sigma^2(X'WX)^{-1}$ is biased.

Variance Estimation by Wolter derives a lot of variance estimators for complex survey data, but doesn't really demonstrate why regular variance estimating techniques are biased, which is what I would like to explain to my professor.

Any help would be appreciated.

Regards

Answer 1

Here are some explicit ways that the model-based estimator can be biased

1. Heteroskedasticity. Let X be binary and Y be continuous. We know that linear regression of Y on X reproduces Student's t-test (the equal-variance t-test), and we know that if the variance of Y is different between the X groups that the t-test has the wrong level. If the smaller group has larger variance, the t-test is anticonservative; if the smaller group has smaller variance, the t-test is conservative. That means the standard error is wrong: too small or too large depending on the group sizes.

Some moderately tedious linear algebra shows that the Satterthwaite/Welch t-statistic is what you get by using the sandwich variance estimator in the regression of Y on X. We know the Welch t-test (in not-too-small samples) has correct size even when the variance of Y differs by X, so it must be using the correct standard error.

2. Pseudoreplication. Suppose you have N observations of X and Y, and you take M identical copies of each one. The model-based variance is too small exactly by a factor of M. The sandwich variance is still correct: there's a factor of M^2 in the middle term and factors of 1/M in each outer term.

3. Precision vs sampling weights. You can think of these as related to the difference between replication and pseudoreplication. The classical derivation of WLS is that an observation (X,Y) with a weight of W arises when you have W independent observations with the same value of X, and you take Y to be the average. It's real replication W times. A sampling weight of W is equivalent to pseudoreplication: you have one observation, but you replicate it W times to correspond to the W individuals in the population it represents. The correlations between residuals for pseudoreplicates are all 1; the correlations between residuals for true replicates are basically 0. The model-based variance estimator treats them the same, but the sandwich estimator doesn't.

Answer 2

A sort of 'proof by contradiction' is readily available on consideration of the scaling laws at work here, in light of information concepts. The usual estimator(s) you cite, since they ignore the correlation structure within the survey instrument, yield variance estimates that scale inversely with the number of survey questions. Thus, doubling the length of the survey (i.e., the number of items) would be thought to halve the variance of $\hat{\beta}$. But it is easy to appreciate that eventually you must run out of interesting (i.e., informative) new questions to ask a respondent, and thus that estimators ignorant of this fact will systematically overestimate the precision of $\hat{\beta}$.

To be clear, I'm advancing this argument on informational grounds, and in particular am making no appeal to such sheer practicalities as respondent fatigue, which are of course irrelevant to the theoretical content of your question. Any survey designer appreciates intuitively that eventually one exhausts the potential for novelty. Whereas one can 'interrogate' a coin repeatedly, with each flip yielding the same amount of information (about the coin's fairness) as every other flip in a sequence, this is not true for people. At some point, you would be able to predict quite accurately an individual's response to question $n+1$ from his/her responses to questions $1,\dots,n$. Thus, the rate at which new information arrives as you administer a survey to an individual person is monotone decreasing and has an asymptote at zero in the limit of an infinitely long questionnaire. Consequently, it is inconceivable that the precision of $\hat{\beta}$ from a survey should scale in the same way (i.e., linearly) as that of $\hat{p}_{heads}$ in a coin-flipping experiment, where the rate of information arrival is constant.