# How to calculate the variance of a partition of variables

I'm running an experiment where I'm gathering (independent) samples in parallel, I compute the variance of each group of samples and now I want to combine then all to find the total variance of all the samples.

I'm having a hard time finding a derivation for this as I'm not sure of terminology. I think of it as a partition of one RV.

So I want to find $Var(X)$ from $Var(X_1)$, $Var(X_2)$, ..., and $Var(X_n)$, where $X$ = $[X_1, X_2, \dots, X_n]$.

EDIT: The partitions are not the same size/cardinality, but the sum of the partition sizes equal the number of samples in the overall sample set.

EDIT 2: There is a formula for a parallel computation here, but it only covers the case of a partition into two sets, not $n$ sets.

-
Is this the same as my question here: mathoverflow.net/questions/64120/… –  user4500 May 6 '11 at 20:29
What does that last bracket mean? And what do you mean by "total variance"? Is it anything other than the variance of the combined dataset? –  whuber May 6 '11 at 20:38
@whuber which last bracket? "total variance" means the variance of the total dataset. –  gallamine May 9 '11 at 12:37
The expression $[X_1, X_2, \dots, X_n]$ could mean many things (although conventionally it would be a vector): I was looking for a clarification. –  whuber May 9 '11 at 13:59

The formula is fairly straightforward if all the sub-sample have the same sample size. If you had $g$ sub-samples of size $k$ (for a total of $gk$ samples), then the variance of the combined sample depends on the mean $E_j$ and variance $V_j$ of each sub-sample: $$Var(X_1,\ldots,X_{gk}) = \frac{k-1}{gk-1}(\sum_{j=1}^g V_j + \frac{k(g-1)}{k-1} Var(E_j)),$$ where by $Var(E_j)$ means the variance of the sample means.

A demonstration in R:

> x <- rnorm(100)
> g <- gl(10,10)
> mns <- tapply(x, g, mean)
> vs <- tapply(x, g, var)
> 9/99*(sum(vs) + 10*var(mns))
[1] 1.033749
> var(x)
[1] 1.033749


If the sample sizes are not equal, the formula is not so nice.

EDIT: formula for unequal sample sizes

If there are $g$ sub-samples, each with $k_j, j=1,\ldots,g$ elements for a total of $n=\sum{k_j}$ values, then $$Var(X_1,\ldots,X_{n}) = \frac{1}{n-1}\left(\sum_{j=1}^g (k_j-1) V_j + \sum_{j=1}^g k_j (\bar{X}_j - \bar{X})^2\right),$$ where $\bar{X} = (\sum_{j=1}^gk_j\bar{X}_j)/n$ is the weighted average of all the means (and equals to the mean of all values).

Again, a demonstration:

> k <- rpois(10, lambda=10)
> n <- sum(k)
> g <- factor(rep(1:10, k))
> x <- rnorm(n)
> mns <- tapply(x, g, mean)
> vs <- tapply(x, g, var)
> 1/(n-1)*(sum((k-1)*vs) + sum(k*(mns-weighted.mean(mns,k))^2))
[1] 1.108966
> var(x)
[1] 1.108966


By the way, these formulas are easy to derive by writing the desired variance as the scaled sum of $(X_{ji}-\bar{X})^2$, then introducing $\bar{X}_j$: $[(X_{ji}-\bar{X}_j)-(\bar{X}_j-\bar{X})]^2$, using the square of difference formula, and simplifying.

-
thanks. Unfortunately, I can't guarantee that my partitions are all the same size. I'm running a massively parallel process where I need to calculate the variances of each partition in parallel then combine in the end, but the results/samples from each parallel process are not equal (it's a Monte Carlo simulation of received photons). –  gallamine May 9 '11 at 12:43
Awesome! Thanks. Also, I believe the summation limit on $\bar{X}$ should be 1 to g, but I can't make 1 letter edits. –  gallamine May 9 '11 at 17:04
You are right, I fixed the formula. –  Aniko May 9 '11 at 17:17
I can't +1 this enough, super helpful formula for parallel computation in a data warehouse environment –  Noah Yetter Mar 8 at 17:31