8
$\begingroup$

A reviewer of mine is asking for a reason why I have used unweighted data, instead of weighted data. I have discussed the issue with a statistician and his response was along the lines of

If you have independent observations and you take the overall mean, its variance is always smaller than the variance from a weighted mean as the estimator. ... So confidence intervals will be widened!

I have since found the following question on this website, and from my understanding, they suggest that the variance should be the same. So can someone, please, with a more statistically gifted mind than mine, please confirm the response from the statistician and explain in layman terms the theory, or with a worked example.

$\endgroup$
  • $\begingroup$ If the "weights" are in fact frequencies of observation or of population, then they must be used, as the unweighted numbers are meaningless. Your statistician's quote is likely to be true for a population with a unimodal distribution, though it need not be true in general. $\endgroup$ – Henry Aug 14 '16 at 15:00
  • $\begingroup$ It would be easy enough to provide a worked example with more context. What do the weights represent? Are you talking about the variance of the sample mean? Are the samples from a finite population? With or without replacement? $\endgroup$ – Henry Aug 14 '16 at 15:02
  • $\begingroup$ Lets say we have collected a series of heart rate measurements from a sample of people in a hospital. A weighting factor can then applied to each individual to scale the measurements to be reflective of national estimates or population - by comparing a series of confounders (eg. age, height, weight, etc). $\endgroup$ – user08041991 Aug 14 '16 at 15:08
  • $\begingroup$ The question to which you link is about frequency weights. Is that what you have? $\endgroup$ – mdewey Aug 14 '16 at 15:29
  • 2
    $\begingroup$ The mean of $n$ values $x_i$ is the weighted mean $\bar x=\sum_iw_ix_i$ with weights $w_i=1/n$. When the $x_i$ are independent, basic rules of variance imply $$\operatorname{Var}(\bar x) =\sum_iw_i^2 \operatorname{Var}(x_i).\tag{1}$$ When in addition the $x_i$ all have the same variance $\sigma^2$, this simplifies to $\sum w_i^2$ times $\sigma^2$. Since weights are positive and sum to unity, $(1)$ is minimized only when $w_i=1/n$. In this sense the statistician is correct. This general conclusion is independent of any other property of the distribution of the $x_i$, such as unimodality. $\endgroup$ – whuber Aug 14 '16 at 18:41
5
$\begingroup$

Your linked question is addressing using weights as a shortcut for dealing with equally weighted per data point variance in which some data points occur more than once.

@whuber has addressed in a comment the situation in which the variances of all data points are equal. So I will address the situation in which they are not equal. It is in this situation that the optimal weighted mean produces a lower variance than the unweighted, i.e., equally weighted, mean.

The weighted mean, using weights $w_i$, equals $\Sigma_{i=1}^n{w_i x_i}$, and has variance = $\Sigma_{i=1}^n{w_i^2 Var(x_i)}$. So we wish to minimize $\Sigma_{i=1}^n{w_i^2 Var(x_i)}$, subject to $\Sigma_{i=1}^n{w_i} = 1$ and $w_i \ge 0$ for all i.

The Karush-Kuhn-Tucker conditions, which are necessary and sufficient for a global minimum for this problem, given that it is a convex Quadratic Programming problem, result in a closed form solution, namely:

The optimal $w_i = [1/Var(x_i)]/\Sigma_{j=1}^n{[1/Var(x_j)]}$ for 1 = 1 .. n.

The variance of the corresponding optimal weighted mean = $1/\Sigma_{i=1}^n{[1/Var(x_i)]}$.

By contrast, equal weighting means $w_i = \frac{1}{n}$ for all i, where n is the number of data points. As pointed out by whuber, equal weights are optimal if all data point variances are equal, which can be seen from the above formula for optimal $w_i$. However, as evident by that formula, equal weights are not optimal if the data point variances are not all equal, and indeed result in larger variance (of the weighted mean) than the optimal weights. The variance of the equally weighted mean, i.e., the variance of the weighted mean using equal weights = $\frac{1}{n^2}\Sigma_{i=1}^n{Var(x_i)}$.

Here are some example numerical results:

  1. There are two data points, having variances respectively of 1 and 4. The unweighted mean has variance = 1.25. The weighted mean using the optimal weights of 0.8 and 0.2 respectively, has variance = 0.8, which of course is less than 1.25.
  2. There are three data points, having variances respectively of 1, 4, and 9. The unweighted mean has variance = 1.5556. The weighted mean using the optimal weights of 0.7347, 0.1837, 0.0816 respectively, has variance = 0.7347, which of course is less than 1.5556.

Of course, it is possible for the weighted mean to have a greater variance than the unweighted mean, if the weights are chosen in a poor manner. By choosing weight of 1 on the data point with largest variance, and 0 for all other data points, the weighted mean would have variance = the largest variance of any data point. This extreme example would be the result of maximizing rather than minimizing in the optimization problem I laid out.

$\endgroup$
  • $\begingroup$ I'm confused about your reference to individual data points having variance (e.g. there are two data points, having variances respectively of 1 and 4), can you please explain? $\endgroup$ – edstatsuser Jul 10 at 5:18
  • $\begingroup$ Saying data point $x_i$ has a particular variance is short-hand for saying that $x_i$ is drawn from a population (random variable) which has that variance. So the different data points can be drawn from different populations, because this is not assumed to be i.i.d. sampling. $\endgroup$ – Mark L. Stone Jul 10 at 10:01
0
$\begingroup$

Here is a simple example using the $\frac1n\sum_i\left(x_i-\frac1n\sum_j x_j\right)^2$ and $\frac1{\sum_k w_k}\sum_i w_i\left(x_i-\frac1{\sum_k w_k}\sum_j w_j x_j\right)^2$ forms of the variance:

Suppose your population has measurements $20,30,40,50$.

  • Unweighted the mean is $35$ and variance is $125$
  • With respective weights $1000,4000,3000,2000$ the weighted mean is $36$ and the weighted variance is $84$
  • With respective weights $3000,2000,1000,4000$ the weighted mean is $36$ and the weighted variance is $164$

This example is consistent with my comment that the your statistician's quote is likely to be true for a population with a unimodal distribution, though it need not be true in general.

I suppose the point is that if you are quoting the weighted mean, you should probably be associating it with the weighted variance. If in fact your mean is the result of the sample, the standard error of the weighted sample mean is a more complicated calculation.

$\endgroup$
  • $\begingroup$ This answer appears to confuse the variance of a sample (or finite population) with the variance of the sampling distribution of the mean (or weighted mean). Consequently it includes statements that appear not to be true and may be misleading. $\endgroup$ – whuber Aug 14 '16 at 18:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.