How to (statistically) test for the difference in the mean of an outcome variable when using weights, vs not using them?

Question

Notations

• $y$ = outcome variable of interest.
• $w$ = weights to use on outcome $y$ (estimated with some method, be it post-stratification, IPW, or something else)
• $\bar y$ = summary statistic of interest (e.g.: the mean of $y$)
• $\bar y^*$ = the weighted version of the mean of $y$ ($\bar y^* = (\sum y_i * w_i) / (\sum w_i)$)

Question

How can we tell if it's "worth it" to use weights with an outcome of interest? I.e.: which is the better estimation of some $\mu$? $\bar y^*$ or $\bar y$?

There is obviously a bias-variance trade-off. Can we estimate it somehow?

Are there common practices/references for methods to check these?

Extra background

In survey methodology there are various ways of producing weights for adjusting for things such as non-response bias. Using these weights offers a tradeoff in which we sacrifice statistical power in exchange for (ideally) less biased estimates. In cases where there is no real correlation between the weights and the outcome of interest, then using the weights would lead to a much less accurate estimate (due to the error produced by the use of the weights), with no real gain in the bias front.

Potential ideas I had

As stated, my interest is in testing if we should use the weights we have or not (ignoring the potential post-selection inference issue that this might cause).

1. Directly check the correlation (be it using paerson, spearman, kendal-tau, or some other one) between the correlation and variable of interest. If the correlation is not statistically significant then there isn't much point in using the weights

2. Use a statistical test to compare if $\bar y^*$ is statistically different then $\bar y$? We could treat the sample with and without the weights as two i.i.d samples and compare them using a t-test. My concern is that it doesn't seem like an i.i.d sample, so my intuition is that a simple t-test wouldn't be valid here. I'm not sure how to adjust for it directly (would love to hear if there is a way!). Alternatively, I imagine we can use a bootstrap method to construct a CI for the difference of $\bar y^* - \bar y$ (each time sampling, with replacement, a pair from y and w, and thus having the empirical distribution of the difference).

Would love to have more ideas / references from others here. Thanks upfront!

Answer 1

For simplicity, I am going to assume that all your observable values have the same mean $\mu$, which is the population parameter of intereset. (It is possible to generalise away from this, but it would require you to be more specific about the population quantity you are estimating.) It is possible to proceed by looking at the best unbiased estimator under general assumptions about the relationship between the observable variables.

As you note in your question, in the standard case where we have IID observations, the unweighted sample mean gives the best estimator of $\mu$. Now, obviously if you want to find a situation where non-equal weightings is optimal, you are going to need to depart from the IID form somehow. Your question does not specify the situation you have in mind, but one useful generalisation is to assume a general covariance structure among the observations. Suppose we let $\mathbb{E}(\mathbf{Y}) = \mu \mathbf{1}$ and $\mathbb{C}(\mathbf{Y}) = \boldsymbol{\Sigma}$ where the latter is some known covariance matrix. In this case we can show that:

$$\mathbb{E}(\bar{Y}^*) = \mu \quad \quad \quad \quad \quad \mathbb{V}(\bar{Y}^*) = \frac{\sum_i \sum_j w_i w_j \sigma_{i,j}}{(\sum w_i)^2}.$$

Any weightings give an unbiased estimator in this case, and to minimise the variance we solve the constrained optimisation problem:

$$\text{Minimise} \ \sum_i \sum_j w_i w_j \sigma_{i,j} \quad \quad \text{subject to} \quad \quad \sum w_i = 1.$$

The Lagrangian function for this optimisation problem is:

$$\mathcal{L}(\mathbf{w}, \lambda) = \sum_i \sum_j w_i w_j \sigma_{i,j} - \lambda \big( \sum w_i - 1 \big).$$

This function has gradient vector and Hessian matrix given respectively by:

$$\begin{align} \nabla_\mathbf{w} \mathcal{L} (\mathbf{w}, \lambda) &= 2 \boldsymbol{\Sigma} \hat{\mathbf{w}} - \lambda \mathbf{1}, \\[10pt] \nabla_\mathbf{w}^2 \mathcal{L} (\mathbf{w}, \lambda) &= 2 \boldsymbol{\Sigma}. \\[6pt] \end{align}$$

We can write the critical point equation as $\boldsymbol{\Sigma} \hat{\mathbf{w}} = \tfrac{\lambda}{2} \mathbf{1}$, which requires us to find the row reduced echelon form for the matrix $[ \ \boldsymbol{\Sigma} \ | \ \tfrac{\lambda}{2} \mathbf{1} \ ]$ (solving for $\lambda$ simultaneously).

Observe that in the simple case where we have uncorrelated homoskedastic observations (including the IID case) we have $\boldsymbol{\Sigma} = \sigma^2 \mathbf{I}$ which gives the critical point equation $\hat{\mathbf{w}} \propto \mathbf{1}$, which is then solved by taking $\hat{w}_1 = \cdots = \hat{w}_n = \tfrac{1}{n}$ (i.e., we minimise the variance by taking equal weights on the observations). Departure from optimisation by using equal weights therefore occurs when we vary $\boldsymbol{\Sigma}$ to some other form. This might entail having some heteroskedasticity in the observations, or some correlation between the observations, or both.

As to your suggestions, I see no value in the suggestion to check if $\bar{Y}^*$ and $\bar{Y}$ are "statistically different". Both of these are known quantities once we observe our data, so whether they are either the same or different is observed. (There is therefore no valid statistical hypothesis test on the difference other that a trivial test.) It is also not possible to check for correlation between the observations unless you are observing multiple instances of the entire vector of observations (e.g., IID instances of a random vector). I think your best bet here is to think about the kinds of covariance structures that you might get for your observations and see if these lead to optimisation using unequal weights.