10
$\begingroup$

A reviewer of mine is asking for a reason why I have used unweighted data, instead of weighted data. I have discussed the issue with a statistician and his response was along the lines of

If you have independent observations and you take the overall mean, its variance is always smaller than the variance from a weighted mean as the estimator. ... So confidence intervals will be widened!

I have since found the following question on this website, and from my understanding, they suggest that the variance should be the same. So can someone, please, with a more statistically gifted mind than mine, please confirm the response from the statistician and explain in layman terms the theory, or with a worked example.

$\endgroup$
6
  • $\begingroup$ If the "weights" are in fact frequencies of observation or of population, then they must be used, as the unweighted numbers are meaningless. Your statistician's quote is likely to be true for a population with a unimodal distribution, though it need not be true in general. $\endgroup$
    – Henry
    Commented Aug 14, 2016 at 15:00
  • $\begingroup$ It would be easy enough to provide a worked example with more context. What do the weights represent? Are you talking about the variance of the sample mean? Are the samples from a finite population? With or without replacement? $\endgroup$
    – Henry
    Commented Aug 14, 2016 at 15:02
  • $\begingroup$ Lets say we have collected a series of heart rate measurements from a sample of people in a hospital. A weighting factor can then applied to each individual to scale the measurements to be reflective of national estimates or population - by comparing a series of confounders (eg. age, height, weight, etc). $\endgroup$ Commented Aug 14, 2016 at 15:08
  • $\begingroup$ The question to which you link is about frequency weights. Is that what you have? $\endgroup$
    – mdewey
    Commented Aug 14, 2016 at 15:29
  • 2
    $\begingroup$ The mean of $n$ values $x_i$ is the weighted mean $\bar x=\sum_iw_ix_i$ with weights $w_i=1/n$. When the $x_i$ are independent, basic rules of variance imply $$\operatorname{Var}(\bar x) =\sum_iw_i^2 \operatorname{Var}(x_i).\tag{1}$$ When in addition the $x_i$ all have the same variance $\sigma^2$, this simplifies to $\sum w_i^2$ times $\sigma^2$. Since weights are positive and sum to unity, $(1)$ is minimized only when $w_i=1/n$. In this sense the statistician is correct. This general conclusion is independent of any other property of the distribution of the $x_i$, such as unimodality. $\endgroup$
    – whuber
    Commented Aug 14, 2016 at 18:41

3 Answers 3

7
$\begingroup$

Your linked question is addressing using weights as a shortcut for dealing with equally weighted per data point variance in which some data points occur more than once.

@whuber has addressed in a comment the situation in which the variances of all data points are equal. So I will address the situation in which they are not equal. It is in this situation that the optimal weighted mean produces a lower variance than the unweighted, i.e., equally weighted, mean.

The weighted mean, using weights $w_i$, equals $\Sigma_{i=1}^n{w_i x_i}$, and has variance = $\Sigma_{i=1}^n{w_i^2 Var(x_i)}$. So we wish to minimize $\Sigma_{i=1}^n{w_i^2 Var(x_i)}$, subject to $\Sigma_{i=1}^n{w_i} = 1$ and $w_i \ge 0$ for all i.

The Karush-Kuhn-Tucker conditions, which are necessary and sufficient for a global minimum for this problem, given that it is a convex Quadratic Programming problem, result in a closed form solution, namely:

The optimal $w_i = [1/Var(x_i)]/\Sigma_{j=1}^n{[1/Var(x_j)]}$ for 1 = 1 .. n.

The variance of the corresponding optimal weighted mean = $1/\Sigma_{i=1}^n{[1/Var(x_i)]}$.

By contrast, equal weighting means $w_i = \frac{1}{n}$ for all i, where n is the number of data points. As pointed out by whuber, equal weights are optimal if all data point variances are equal, which can be seen from the above formula for optimal $w_i$. However, as evident by that formula, equal weights are not optimal if the data point variances are not all equal, and indeed result in larger variance (of the weighted mean) than the optimal weights. The variance of the equally weighted mean, i.e., the variance of the weighted mean using equal weights = $\frac{1}{n^2}\Sigma_{i=1}^n{Var(x_i)}$.

Here are some example numerical results:

  1. There are two data points, having variances respectively of 1 and 4. The unweighted mean has variance = 1.25. The weighted mean using the optimal weights of 0.8 and 0.2 respectively, has variance = 0.8, which of course is less than 1.25.
  2. There are three data points, having variances respectively of 1, 4, and 9. The unweighted mean has variance = 1.5556. The weighted mean using the optimal weights of 0.7347, 0.1837, 0.0816 respectively, has variance = 0.7347, which of course is less than 1.5556.

Of course, it is possible for the weighted mean to have a greater variance than the unweighted mean, if the weights are chosen in a poor manner. By choosing weight of 1 on the data point with largest variance, and 0 for all other data points, the weighted mean would have variance = the largest variance of any data point. This extreme example would be the result of maximizing rather than minimizing in the optimization problem I laid out.

$\endgroup$
2
  • $\begingroup$ I'm confused about your reference to individual data points having variance (e.g. there are two data points, having variances respectively of 1 and 4), can you please explain? $\endgroup$ Commented Jul 10, 2019 at 5:18
  • $\begingroup$ Saying data point $x_i$ has a particular variance is short-hand for saying that $x_i$ is drawn from a population (random variable) which has that variance. So the different data points can be drawn from different populations, because this is not assumed to be i.i.d. sampling. $\endgroup$ Commented Jul 10, 2019 at 10:01
1
$\begingroup$

Here is a simple example using the $\frac1n\sum_i\left(x_i-\frac1n\sum_j x_j\right)^2$ and $\frac1{\sum_k w_k}\sum_i w_i\left(x_i-\frac1{\sum_k w_k}\sum_j w_j x_j\right)^2$ forms of the variance:

Suppose your population has measurements $20,30,40,50$.

  • Unweighted the mean is $35$ and variance is $125$
  • With respective weights $1000,4000,3000,2000$ the weighted mean is $36$ and the weighted variance is $84$
  • With respective weights $3000,2000,1000,4000$ the weighted mean is $36$ and the weighted variance is $164$

This example is consistent with my comment that the your statistician's quote is likely to be true for a population with a unimodal distribution, though it need not be true in general.

I suppose the point is that if you are quoting the weighted mean, you should probably be associating it with the weighted variance. If in fact your mean is the result of the sample, the standard error of the weighted sample mean is a more complicated calculation.

$\endgroup$
1
  • $\begingroup$ This answer appears to confuse the variance of a sample (or finite population) with the variance of the sampling distribution of the mean (or weighted mean). Consequently it includes statements that appear not to be true and may be misleading. $\endgroup$
    – whuber
    Commented Aug 14, 2016 at 18:35
0
$\begingroup$

Searching for how to compute variance of a weighted sum I came across this question, and I don't see a satisfactory answer that applies to the situation state by @user08041991, namely "Let's say we have collected a series of heart rate measurements from a sample of people in a hospital. A weighting factor can then applied to each individual to scale the measurements to be reflective of national estimates."

In abstract terms, let $\{x_1,\dots,x_L\}$ be an i.i.d. sample drawn from a random variable $X$ with distribution $dP$ (in the example, these are the observed values of heart rate of people in a hospital). We want to estimate the expected value of $X$ not under $dP$, but under another probability distribution, $\omega\ dP$, where $\omega > 0$ a.s. [$dP$] is a known "weight" function, and the values ${\omega_1,\dots,\omega_L}$ at the randomly sampled observations are known. The setting bares resemblance to setting of importance sampling, where the weight is sought.

We can use the observations of $X$ and $\omega$ to define an estimator of the expected value of $X$ with respect to $\omega\ dP$, namely, the weighted average: $$ \bar{x} = \frac{\sum_{j=1}^L x_j\omega_j}{ \sum_{j=1}^L \omega_j} $$

The exact mean and variance of $\bar{x}$ cannot be given in closed form in general. However, one can compute the asymptotic distribution of $\bar{x}$ as $L \rightarrow \infty$. Towards that goal, consider the random vector $$ V_L = \left(\begin{matrix} \frac{1}{L} \sum_j x_j \omega_j \\ \frac{1}{L} \sum_j \omega_j \end{matrix} \right) $$ which, under suitable assumptions. converges a.e.[$dP$] to $$ \left(\begin{matrix} \int{X \omega\ dP} \\ 1 \end{matrix} \right) ; $$

moreover, under suitable assumptions, $$ \sqrt{L} \left( V_L - \left(\begin{matrix} \int{X \omega\ dP} \\ 1 \end{matrix} \right) \right) \overset{P}{\rightsquigarrow} Z $$

where $Z = \left(\begin{matrix} z_1 \\ z_2 \end{matrix} \right)$ is a normal vector with mean 0 and covariance $$ \Sigma = \left[\begin{matrix} Var_P(X \omega) & Cov_P(X \omega, \omega) \\ Cov_P(X \omega, \omega) & Var_P(\omega) \end{matrix} \right] $$ (here $Var_P$ is the variance under probability $dP$, etc.)

Therefore,

$$ \bar{x} = \frac{ \frac{1}{L} \sum_j x_j \omega_j } {\frac{1}{L} \sum_j \omega_j} = \frac{ \int{X \omega\ dP} + \frac{1}{ \sqrt{L}} z_1}{ 1 + \frac{1}{\sqrt{L}} z_2} \\ \approx \int{X \omega\ dP} + \frac{1}{ \sqrt{L}} (z_1 -z_2) \\ = \int{X \omega\ dP} + \frac{1}{ \sqrt{L}} (1, -1)Z $$

Now $(1, -1)Z$ is normal, with mean $0$ and variance $$ E( (1,-1)Z Z^* (\begin{matrix} 1 \\-1 \end{matrix} )) = (1,-1) \Sigma (\begin{matrix} 1 \\-1 \end{matrix} ) \\ = Var_P(X \omega) - 2 Cov_P(X \omega, \omega) + Var_P(\omega) \\ = Var_P( X \omega - \omega) = Var_P( (X-1)\omega) $$

Thus, we get: $$ \sqrt{L} \Big( \bar{x} - \int{X \omega\ dP} \Big) ~ \overset{dP}{\rightsquigarrow } ~ u $$ where $u$ is normally distributed, with mean $0$ and variance $Var_P\left( (X-1)\omega \right) $. The variance of $u$ can be estimated with the sample variance of $(X-1)\omega$.

A couple of notes on the original question:

  • the unweighted mean would not be an estimator of $\int{X \omega\ dP}$ in general, so if the objective is to estimate a quantity in a general population by means of observations in a different population, the consideration of simple mean is irrelevant ( $\omega$ is the likelihood ratio of the general population likelihood, and the observed population likelihood)

  • If one smooths $X$ by averaging over some other variable $A$ (e.g.: averaging over age group) to get $E(X|A)$, then the variance of the result is reduced: $Var( E(X|A) ) \le Var(X)$, what might have been what the statistician you consulted had in mind. The case of your question is different. In general, for a given $X$, $Var( (X-1)\omega)$ can be larger or smaller than $Var( (X-1) ) $, depending on $\omega$ (example follows).

** Example ** Here is an example to show that when the weights are case specific, the (asymptotic) standard deviation of the weighted mean is not comparable to the sd. of the unweighted mean. We work on the real line. Let $\varphi(x) = \frac{1}{2}$ if $|x| < 1$, $0$ otherwise and let $$ dP = \frac{1}{2} \left[ \frac{1}{\epsilon} \varphi(\frac{x}{\epsilon} ) + \varphi(x-4) \right] dx ; $$ $dP$ is a probability measure on the real line. We will take $\epsilon > 0$ and small.

Let $ \omega(x) = A$ if $-1 < x < 1$, $B$ if $1 \le x < 5$, where $A,B > 0$. Then $\omega(x) > 0$ a.e. $[dP]$. Moreover, $\int \omega dP = \frac{A+B}{2} = 1$ if we take $A,B$ so that $A+B=2$. The choice $A=B=1$ gives $\omega(x) = 1$, and using these values corresponds to using unweighted means.

When $A=1 - \delta$, $B=1+\delta$, a staightfoward computation shows that $Var( (X-1)\omega )$ is a quadratic function of $\delta$ of the form: $$ Var( (X-1)\omega ) = Var(X) + 2\left[ \frac{7-\epsilon^2}{3} \right]\ \delta + O(\delta^2) $$
so for small $\delta$, $Var( (X-1)\omega) > Var(X)$ if $\delta > 0$ and $Var( (X-1)\omega) < Var(X)$ if $\delta < 0$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.