# How to calculate the sample variance of aggregated data

I have dataset where I am only seeing the aggregated data. Assume that there is an underlying random variable, $p$. The sample data is of the form $P_i = \sum^{n_i}_{j=1} p_{ij}$ with $i=1..N$. The individual $n_i$ are randomly distributed. I would like to estimate the variance of $p$ from this dataset.

In my case, about one third of the data has $n_i = 1$, so a simple approach is to use only this subset of the data, and estimate the sample variance of $p$ directly from it. However, this is discarding two thirds of the data. Is there a better method of calculating the sample variance using all of the data?

• Unfortunately I do not have the sample variance of the groups. Oct 20, 2015 at 10:23
• I am not so sure that you cannot recover the variance. My example above of choosing all groups with $n_i =1$ does provide a method, although only for data sets that have samples with $n_i = 1$. Extending this idea further, one approach would be to divide all the samples into groups with the same value of $n_i$. You can calculate the sample variance of each group. The shape of the graph of sample variance versus $n_i$ must have some information about the underlying variance. Oct 20, 2015 at 10:30

We can solve this with likelihood methods if we, in addition, can assume the original individual observations are iid normal. So suppose you have $$x_{i,1},x_{i,2},\dots, x_{i,n_i} \sim \text{N}(\mu, \sigma^2)$$ independently, but you do only know $$\bar{x}_i \sim \text{N}(\mu, \sigma^2/n_i), \quad, i=1,2\dots,m$$ where some of the (known) $n_i$'s are equal to 1.
Write the likelihood function, take logarithms and you find the loglikelihood function as $$\ell = \sum_{i=1}^m \cdot \log \sqrt{n_i} - \frac12 \log (2\pi\sigma^2) -\frac12 n_i \left(\frac{\bar{x}_i-\mu}{\sigma}\right)^2$$ differentiating this with respect to $\mu$, setting equal to zero gives the likelihood equation for $\mu$, with solution $$\hat{\mu}=\frac{\sum n_i \bar{x}_i}{\sum n_i}$$ that is, the solution is the weighted least squares estimator.
To find an estimator for the variance $\sigma^2$, differentiate the loglikelihood function with respect to $\sigma^2$: First write $$S^2 = \sum n_i (\bar{x}_i -\hat{\mu})^2$$ Then the concentrated loglikelihood can be written $$\ell^* = \frac12\sum \log(n_i) -\frac{m}{2}\log(2\pi \sigma^2) - \frac12\frac{S^2}{\sigma^2}$$ differentiating, setting equal to zero and solving gives the estimator of the variance as $$\hat{\sigma^2} = \frac{S^2}{m}$$ The same approach can be used if the parent distribution is something other than normal.