# How to calculate the variance of a partition of variables

I'm running an experiment where I'm gathering (independent) samples in parallel, I compute the variance of each group of samples and now I want to combine then all to find the total variance of all the samples.

I'm having a hard time finding a derivation for this as I'm not sure of terminology. I think of it as a partition of one RV.

So I want to find $Var(X)$ from $Var(X_1)$, $Var(X_2)$, ..., and $Var(X_n)$, where $X$ = $[X_1, X_2, \dots, X_n]$.

EDIT: The partitions are not the same size/cardinality, but the sum of the partition sizes equal the number of samples in the overall sample set.

EDIT 2: There is a formula for a parallel computation here, but it only covers the case of a partition into two sets, not $n$ sets.

• Is this the same as my question here: mathoverflow.net/questions/64120/… – user4500 May 6 '11 at 20:29
• What does that last bracket mean? And what do you mean by "total variance"? Is it anything other than the variance of the combined dataset? – whuber May 6 '11 at 20:38
• @whuber which last bracket? "total variance" means the variance of the total dataset. – gallamine May 9 '11 at 12:37
• The expression $[X_1, X_2, \dots, X_n]$ could mean many things (although conventionally it would be a vector): I was looking for a clarification. – whuber May 9 '11 at 13:59

The formula is fairly straightforward if all the sub-sample have the same sample size. If you had $g$ sub-samples of size $k$ (for a total of $gk$ samples), then the variance of the combined sample depends on the mean $E_j$ and variance $V_j$ of each sub-sample: $$Var(X_1,\ldots,X_{gk}) = \frac{k-1}{gk-1}(\sum_{j=1}^g V_j + \frac{k(g-1)}{k-1} Var(E_j)),$$ where by $Var(E_j)$ means the variance of the sample means.

A demonstration in R:

> x <- rnorm(100)
> g <- gl(10,10)
> mns <- tapply(x, g, mean)
> vs <- tapply(x, g, var)
> 9/99*(sum(vs) + 10*var(mns))
[1] 1.033749
> var(x)
[1] 1.033749


If the sample sizes are not equal, the formula is not so nice.

EDIT: formula for unequal sample sizes

If there are $g$ sub-samples, each with $k_j, j=1,\ldots,g$ elements for a total of $n=\sum{k_j}$ values, then $$Var(X_1,\ldots,X_{n}) = \frac{1}{n-1}\left(\sum_{j=1}^g (k_j-1) V_j + \sum_{j=1}^g k_j (\bar{X}_j - \bar{X})^2\right),$$ where $\bar{X} = (\sum_{j=1}^gk_j\bar{X}_j)/n$ is the weighted average of all the means (and equals to the mean of all values).

Again, a demonstration:

> k <- rpois(10, lambda=10)
> n <- sum(k)
> g <- factor(rep(1:10, k))
> x <- rnorm(n)
> mns <- tapply(x, g, mean)
> vs <- tapply(x, g, var)
> 1/(n-1)*(sum((k-1)*vs) + sum(k*(mns-weighted.mean(mns,k))^2))
[1] 1.108966
> var(x)
[1] 1.108966


By the way, these formulas are easy to derive by writing the desired variance as the scaled sum of $(X_{ji}-\bar{X})^2$, then introducing $\bar{X}_j$: $[(X_{ji}-\bar{X}_j)-(\bar{X}_j-\bar{X})]^2$, using the square of difference formula, and simplifying.

• thanks. Unfortunately, I can't guarantee that my partitions are all the same size. I'm running a massively parallel process where I need to calculate the variances of each partition in parallel then combine in the end, but the results/samples from each parallel process are not equal (it's a Monte Carlo simulation of received photons). – gallamine May 9 '11 at 12:43
• I can't +1 this enough, super helpful formula for parallel computation in a data warehouse environment – Noah Yetter Mar 8 '13 at 17:31

This is simply an add-on to the answer of aniko with a rough sketch of the derivation and some python code, so all credits go to aniko.

# derivation

Let $$X_j \in X = \{X_1, X_2, \ldots, X_g\}$$ be one of $$g$$ parts of the data where the number of elements in each part is $$k_j = |X_j|$$. We define the mean and the variance of each part to be \begin{align*} E_j & = \mathrm{E}\left[X_j\right] = \frac{1}{k_j} \sum_{i=1}^{k_j} X_{ji}\\ V_j & = \mathrm{Var}\left[X_j\right] = \frac{1}{k_j-1} \sum_{i=1}^{k_j} (X_{ji} - E_j)^2 \end{align*} respectively. If we set $$n = \sum_{j=1}^g k_j$$, the variance of the total dataset is given by: \begin{align*} \mathrm{Var}\left[X\right] & = \frac{1}{n-1} \sum_{j=1}^{g} \sum_{i=1}^{k_j} (X_{ji} - \mathrm{E}\left[X\right])^2 \\ & = \frac{1}{n-1} \sum_{j=1}^{g} \sum_{i=1}^{k_j} \big((X_{ji} - E_j) - (\mathrm{E}\left[X\right] - E_j)\big)^2 \\ & = \frac{1}{n-1} \sum_{j=1}^{g} \sum_{i=1}^{k_j} (X_{ji} - E_j)^2 - 2(X_{ji} - E_j)(\mathrm{E}\left[X\right] - E_j) + (\mathrm{E}\left[X\right] - E_j)^2 \\ & = \frac{1}{n-1} \sum_{j=1}^{g} (k_j - 1) V_j + k_j (\mathrm{E}\left[X\right] - E_j)^2. \end{align*} If we have the same size $$k$$ for each part, i.e. $$\forall j: k_j = k$$, above formula simplifies to \begin{align*} \mathrm{Var}\left[X\right] & = \frac{1}{n-1} \sum_{j=1}^g (k-1) V_j + k(g-1) \mathrm{Var}\left[E_j\right] \\ & = \frac{k-1}{n-1} \sum_{j=1}^g V_j + \frac{k(g-1)}{k-1} \mathrm{Var}\left[E_j\right] \end{align*}

# python code

The following python function works for arrays that have been splitted along the first dimension and implements the "more complex" formula for differently sized parts.

import numpy as np

def combine(averages, variances, counts, size=None):
"""
Combine averages and variances to one single average and variance.

# Arguments
averages: List of averages for each part.
variances: List of variances for each part.
counts: List of number of elements in each part.
size: Total number of elements in all of the parts.
# Returns
average: Average over all parts.
variance: Variance over all parts.
"""
average = np.average(averages, weights=counts)

# necessary for correct variance in case of multidimensional arrays
if size is not None:
counts = counts * size // np.sum(counts, dtype='int')

squares = (counts - 1) * variances + counts * (averages - average)**2
return average, np.sum(squares) / (size - 1)


It can be used as follows:

# sizes k_j and n
ks = np.random.poisson(10, 10)
n = np.sum(ks)

# create data
x = np.random.randn(n, 20)
parts = np.split(x, np.cumsum(ks[:-1]))

# compute statistics on parts
ms = [np.mean(p) for p in parts]
vs = [np.var(p, ddof=1) for p in parts]

# combine and compare
combined = combine(ms, vs, ks, x.size)
numpied = np.mean(x), np.var(x, ddof=1)
distance = np.abs(np.array(combined) - np.array(numpied))
print('combined --- mean:{: .9f} - var:{: .9f}'.format(*combined))
print('numpied  --- mean:{: .9f} - var:{: .9f}'.format(*numpied))
print('distance --- mean:{: .5e} - var:{: .5e}'.format(*distance))

• I believe you need a pair of brackets in your final equation (same size k) after you extract a factor of $k-1$. – Dzamo Norton Sep 29 at 12:08
• @DzamoNorton I am afraid I do not understand where you want additional brackets – Mr Tsjolder Sep 30 at 5:43
• I've illustrated it here mathb.in/45578. You can also see said brackets in the same-size-k formula in the other answer to this question. – Dzamo Norton Sep 30 at 9:19
• But I also see now I'd lost track of the scope of the summand. Now that I see the summand must run to the end of the line, the brackets are indeed unnecessary. Apologies! – Dzamo Norton Sep 30 at 9:27