How to decide if to use weights or not when estimating some $\mu$ of a population that has sub-populations with different $\mu_i$?

Question

Setting and Notation

Let's assume we have a population with (for example) two sub-population. Say, males and females. In the population they are split 50%-50%. We care about the population level parameter $\mu$. Furthermore, the male and female sub-population have their own $\mu_m$ and $\mu_f$, were: $\mu = \frac{1}{2}\mu_m + \frac{1}{2}\mu_f$.

We have a panel of $n$ people, out of which (let's say) 90% are males and 10% are female (i.e.: $n_m = 0.9*n$ and $n_f = 0.1*n$). For each person $i$ in the panel we measure $y_i$. If the panel was fully i.i.d then $E[y_i] = \mu$. However, since we have a bias panel (for example, from non-response bias), then $E[y_i] \neq \mu$. However, even for the biased sample, we know that for all $i$ that are males $E[y_i] = \mu_m$ (and similarly for $i$ that are female $E[y_i] = \mu_f$).

We now have two potential options for estimating $\mu$:

• $\bar y$: which will be a biased estimator of $\mu$ (let's say $E[\bar y] = \mu + B$ (B for bias))
• $\bar y^* = \bar y_m * 0.5 + \bar y_f * 0.5$: which will be an un-biased estimator of $\mu$, but will have much larger variance.

Question

How can we tell which of the two estimators are better to use, $\bar y^*$ or $\bar y$, given that we don't know the real values of $\mu$, $\mu_m$, or $\mu_f$?

There is obviously a bias-variance trade-off. Can we estimate it somehow?

Are there common practices/references for methods to check these?

Potential ideas I had

We can decide to use the estimator that will have the minimal MSE. Of course we do not know what it is for each estimator, so we can decide to estimate it.

For example by saying that:

• $MSE(\bar y) = var(\bar y) + bias^2(\bar y) = var(\bar y) + (\mu + B)^2$
• $MSE(\bar y^*) = var(\bar y^*) + bias^2(\bar y^*)$

We assume that $bias^2(\bar y^*) = 0$, hence: $E[\bar y^*] = 0$. Therefore, if we look at: $\hat B = \bar y - \bar y^*$, we notice that: $E[\bar y - \bar y^*] = E[\bar y] - E[\bar y^*] = \mu + B - \mu = B$. So $\hat B$ is a consistent estimator (via method of moments) for $B$. Hence, $\hat B^2$ will also be a consistent (although probably biased) estimator for $B^2$

With this in mind, we can invoke CLT and the delta method, and build an asymptotic distribution for estimating $D = MSE(\bar y) - MSE(\bar y^*)$: $\hat D = var(\bar y) + (\bar y - \bar y^*)^2 - var(\bar y^*)$. The variance: $Var \left[ var(\bar y) + (\bar y - \bar y^*)^2 - var(\bar y^*) \right]$, will probably need to be estimated via bootstrap. And since $\hat D$ will asymptotically have normal distribution, we can perform an hypothesis test to check if it's larger then 0 ($H_1$) or not ($H_0$). This would give us input if to use $\bar y$ or $\bar y^*$.

Would love to have more ideas / references from others here. Thanks upfront!

Answer 1

Your question proposes fixed sample sizes, but I note that it is possible to optimise the sample sizes for stratified sampling using the method discussed in this related question. In order to subsume both your proposed estimators, let's consider the general estimator of the form:

$$\hat{\mu} = \lambda \bar{y}_M + (1-\lambda) \bar{y}_F,$$

where we choose the weighting $0 \leqslant \lambda \leqslant 1$. The mean and variance of this estimator are:

$$\mathbb{E}(\hat{\mu}) = \lambda \mu_M + (1-\lambda) \mu_F \quad \quad \quad \quad \quad \mathbb{V}(\hat{\mu}) = \lambda^2 \cdot \frac{\sigma_M^2}{n_M} + (1-\lambda)^2 \cdot \frac{\sigma_F^2}{n_F}.$$

Consequently, the mean-squared error (MSE) of the estimator is:

$$\begin{align} \text{MSE}(\hat{\mu}, \mu) &= \mathbb{V}(\hat{\mu}) + \text{Bias}(\hat{\mu}, \mu)^2 \\[12pt] &= \Bigg( \lambda^2 \cdot \frac{\sigma_M^2}{n_M} + (1-\lambda)^2 \cdot \frac{\sigma_F^2}{n_F} \Bigg) + \Bigg( (\lambda-\tfrac{1}{2}) \mu_M + (\tfrac{1}{2}-\lambda) \mu_F \Bigg)^2 \\[6pt] &= \Bigg( \lambda^2 \cdot \frac{\sigma_M^2}{n_M} + (1-\lambda)^2 \cdot \frac{\sigma_F^2}{n_F} \Bigg) + \Bigg( (\lambda-\tfrac{1}{2}) (\mu_M - \mu_F) \Bigg)^2 \\[6pt] &= \lambda^2 \cdot \frac{\sigma_M^2}{n_M} + (1-\lambda)^2 \cdot \frac{\sigma_F^2}{n_F} + (\lambda-\tfrac{1}{2})^2 (\mu_M - \mu_F)^2 \\[6pt] &= \lambda^2 \cdot \frac{\sigma_M^2}{n_M} + (\lambda^2 - 2 \lambda + 1) \cdot \frac{\sigma_F^2}{n_F} + (\lambda^2 - \lambda + \tfrac{1}{4}) (\mu_M - \mu_F)^2 \\[6pt] &= \lambda^2 \cdot \Bigg( \frac{\sigma_M^2}{n_M} + \frac{\sigma_F^2}{n_F} + (\mu_M - \mu_F)^2\Bigg) - 2 \lambda \cdot \Bigg( \frac{\sigma_F^2}{n_F} + \frac{(\mu_M - \mu_F)^2}{2} \Bigg) \\[6pt] &\quad \quad + \Bigg( \frac{\sigma_F^2}{n_F} + \Big( \frac{\mu_M - \mu_F}{2} \Big)^2 \Bigg). \\[6pt] \end{align}$$

This function is a convex quadratic form in $\lambda$, and we can use ordinary calculus methods to minimise it with respect to this parameter. We have first and second derivatives:

$$\begin{align} \frac{\partial \text{MSE}}{\partial \lambda} (\hat{\mu}, \mu) &= 2 \lambda \cdot \Bigg( \frac{\sigma_M^2}{n_M} + \frac{\sigma_F^2}{n_F} + (\mu_M - \mu_F)^2 \Bigg) - 2 \cdot \Bigg( \frac{\sigma_F^2}{n_F} + \frac{(\mu_M - \mu_F)^2}{2} \Bigg), \\[10pt] \frac{\partial^2 \text{MSE}}{(\partial \lambda)^2} (\hat{\mu}, \mu) &= 2 \cdot \Bigg( \frac{\sigma_M^2}{n_M} + \frac{\sigma_F^2}{n_F} + (\mu_M - \mu_F)^2\Bigg), \\[10pt] \end{align}$$

which gives the minimising value:

$$\lambda_* = \frac{\sigma_F^2/n_F + (\mu_M - \mu_F)^2/2}{\sigma_M^2/n_M + \sigma_F^2/n_F + (\mu_M - \mu_F)^2}.$$

If $\sigma_M = \sigma_F$ then we get the value $\lambda_* = \tfrac{1}{2}$, which then gives your estimator $\bar{y}_*$. This weighted estimator is "optimal" when the standard deviations for the two groups are the same. Now, obviously we don't know the true parameters, so we don't know the true MSE minimising value $\lambda_*$. Of course, one might reasonably take its empirical sample estimator, which is:

$$\hat{\lambda}_* = \frac{s_F^2/n_F + (\bar{y}_M - \bar{y}_F)^2/2}{s_M^2/n_M + s_F^2/n_F + (\bar{y}_M - \bar{y}_F)^2}.$$

Now, this method is more general than what you are proposing, since your proposal focuses on only two allowable weightings. My view is that it is sub-optimal to restrict yourself to these two weightings --- if you are going to allow a choice of weightings in the first place then it is better to allow any weighting you want and then develop an appropriate optimisation procedure. Developing a procedure that bifurcates between two sub-optimal weightings is likely to give you a poor weighting in most cases (though your estimator might still be okay, since a sub-optimal weighting is not fatal).