How to (statistically) test for the difference in the mean of an outcome variable when using weights, vs not using them?

Notations

• $$y$$ = outcome variable of interest.
• $$w$$ = weights to use on outcome $$y$$ (estimated with some method, be it post-stratification, IPW, or something else)
• $$\bar y$$ = summary statistic of interest (e.g.: the mean of $$y$$)
• $$\bar y^*$$ = the weighted version of the mean of $$y$$ ($$\bar y^* = (\sum y_i * w_i) / (\sum w_i)$$)

Question

How can we tell if it's "worth it" to use weights with an outcome of interest? I.e.: which is the better estimation of some $$\mu$$? $$\bar y^*$$ or $$\bar y$$?

There is obviously a bias-variance trade-off. Can we estimate it somehow?

Are there common practices/references for methods to check these?

Extra background

In survey methodology there are various ways of producing weights for adjusting for things such as non-response bias. Using these weights offers a tradeoff in which we sacrifice statistical power in exchange for (ideally) less biased estimates. In cases where there is no real correlation between the weights and the outcome of interest, then using the weights would lead to a much less accurate estimate (due to the error produced by the use of the weights), with no real gain in the bias front.

As stated, my interest is in testing if we should use the weights we have or not (ignoring the potential post-selection inference issue that this might cause).

1. Directly check the correlation (be it using paerson, spearman, kendal-tau, or some other one) between the correlation and variable of interest. If the correlation is not statistically significant then there isn't much point in using the weights

2. Use a statistical test to compare if $$\bar y^*$$ is statistically different then $$\bar y$$? We could treat the sample with and without the weights as two i.i.d samples and compare them using a t-test. My concern is that it doesn't seem like an i.i.d sample, so my intuition is that a simple t-test wouldn't be valid here. I'm not sure how to adjust for it directly (would love to hear if there is a way!). Alternatively, I imagine we can use a bootstrap method to construct a CI for the difference of $$\bar y^* - \bar y$$ (each time sampling, with replacement, a pair from y and w, and thus having the empirical distribution of the difference).

Would love to have more ideas / references from others here. Thanks upfront!

• Regarding the wording, I guess you test for a difference (in population or the data generating process), not a statistically significant difference? Commented Dec 22, 2020 at 8:48
• True, I'll try to rephrase. Commented Dec 22, 2020 at 9:34
• May I suggest you to expand a bit this would lead to a much less accurate estimate (due to the error produced by the use of the weights) - an argument, proof or example, so that the statement be somehow supported. Commented Dec 27, 2020 at 12:42
• IPW = iterative proportional (rim aka raking) weighting, or what is it? Commented Dec 27, 2020 at 12:43
• Commented Dec 27, 2020 at 13:26

For simplicity, I am going to assume that all your observable values have the same mean $$\mu$$, which is the population parameter of intereset. (It is possible to generalise away from this, but it would require you to be more specific about the population quantity you are estimating.) It is possible to proceed by looking at the best unbiased estimator under general assumptions about the relationship between the observable variables.

As you note in your question, in the standard case where we have IID observations, the unweighted sample mean gives the best estimator of $$\mu$$. Now, obviously if you want to find a situation where non-equal weightings is optimal, you are going to need to depart from the IID form somehow. Your question does not specify the situation you have in mind, but one useful generalisation is to assume a general covariance structure among the observations. Suppose we let $$\mathbb{E}(\mathbf{Y}) = \mu \mathbf{1}$$ and $$\mathbb{C}(\mathbf{Y}) = \boldsymbol{\Sigma}$$ where the latter is some known covariance matrix. In this case we can show that:

$$\mathbb{E}(\bar{Y}^*) = \mu \quad \quad \quad \quad \quad \mathbb{V}(\bar{Y}^*) = \frac{\sum_i \sum_j w_i w_j \sigma_{i,j}}{(\sum w_i)^2}.$$

Any weightings give an unbiased estimator in this case, and to minimise the variance we solve the constrained optimisation problem:

$$\text{Minimise} \ \sum_i \sum_j w_i w_j \sigma_{i,j} \quad \quad \text{subject to} \quad \quad \sum w_i = 1.$$

The Lagrangian function for this optimisation problem is:

$$\mathcal{L}(\mathbf{w}, \lambda) = \sum_i \sum_j w_i w_j \sigma_{i,j} - \lambda \big( \sum w_i - 1 \big).$$

This function has gradient vector and Hessian matrix given respectively by:

\begin{align} \nabla_\mathbf{w} \mathcal{L} (\mathbf{w}, \lambda) &= 2 \boldsymbol{\Sigma} \hat{\mathbf{w}} - \lambda \mathbf{1}, \\[10pt] \nabla_\mathbf{w}^2 \mathcal{L} (\mathbf{w}, \lambda) &= 2 \boldsymbol{\Sigma}. \\[6pt] \end{align}

We can write the critical point equation as $$\boldsymbol{\Sigma} \hat{\mathbf{w}} = \tfrac{\lambda}{2} \mathbf{1}$$, which requires us to find the row reduced echelon form for the matrix $$[ \ \boldsymbol{\Sigma} \ | \ \tfrac{\lambda}{2} \mathbf{1} \ ]$$ (solving for $$\lambda$$ simultaneously).

Observe that in the simple case where we have uncorrelated homoskedastic observations (including the IID case) we have $$\boldsymbol{\Sigma} = \sigma^2 \mathbf{I}$$ which gives the critical point equation $$\hat{\mathbf{w}} \propto \mathbf{1}$$, which is then solved by taking $$\hat{w}_1 = \cdots = \hat{w}_n = \tfrac{1}{n}$$ (i.e., we minimise the variance by taking equal weights on the observations). Departure from optimisation by using equal weights therefore occurs when we vary $$\boldsymbol{\Sigma}$$ to some other form. This might entail having some heteroskedasticity in the observations, or some correlation between the observations, or both.

As to your suggestions, I see no value in the suggestion to check if $$\bar{Y}^*$$ and $$\bar{Y}$$ are "statistically different". Both of these are known quantities once we observe our data, so whether they are either the same or different is observed. (There is therefore no valid statistical hypothesis test on the difference other that a trivial test.) It is also not possible to check for correlation between the observations unless you are observing multiple instances of the entire vector of observations (e.g., IID instances of a random vector). I think your best bet here is to think about the kinds of covariance structures that you might get for your observations and see if these lead to optimisation using unequal weights.

• Hey Ben, thanks for the detailed response. Some thoughts: first, the situation I had in mind is one in which there is a population level $\mu$ and that is the quantity we wish to estimate. It is likely that each sub population would have their own $\mu_{sp_i}$ and that the correct estimation is the weighted average of these values across their proportion in the population. For example, let's say our only two populations are of males and females. They consist of 50%-50% in the population, but in our sample they happened to be 10%-90% (in favor of males, let's say). Commented Dec 29, 2020 at 14:08
• In such a case, we'd use weights to give the male and female respondents different proportion, doing so would naturally reduce the impact of most males on the estimates, thus increasing the variance of our estimator of the global $\mu$. The question is does this increase in variance "worth" the gain in bias-correction we obtain from using the weights. Commented Dec 29, 2020 at 14:10
• So we can think of having $\mu_m$ for males and $\mu_f$ for females. And if $\mu_m - \mu_f \simeq 0$ then there is no real point in using the weights. But if $|\mu_m - \mu_f| >> 0$ then clearly there is value in using the weights. Commented Dec 29, 2020 at 14:12
• Given the input I provided - do you have a better sense of how to help direct me? Thanks! Commented Dec 29, 2020 at 14:12
• @Tal Galili: I know exactly what to do there, and as luck would have it, that is an example I used to teach in my sampling theory class. So that I am not giving a huge long answer in comments, may I suggest you ask a new question for this specific case (the males/females subgroup inference) and ping me for an answer.
– Ben
Commented Dec 29, 2020 at 21:21