How to calculate the sample variance of aggregated data I have dataset where I am only seeing the aggregated data. Assume that there is an underlying random variable, $p$. The sample data is of the form $P_i = \sum^{n_i}_{j=1} p_{ij}$ with $i=1..N$.  The individual $n_i$ are randomly distributed. I would like to estimate the variance of $p$ from this dataset.
In my case, about one third of the data has $n_i = 1$, so a simple approach is to use only this subset of the data, and estimate the sample variance of $p$ directly from it.  However, this is discarding two thirds of the data. Is there a better method of calculating the sample variance using all of the data?
 A: We can solve this with likelihood methods if we, in addition, can assume the original individual observations are iid normal. So suppose you have 
$$
  x_{i,1},x_{i,2},\dots, x_{i,n_i} \sim \text{N}(\mu, \sigma^2)
$$ independently, but you do only know
$$
   \bar{x}_i \sim \text{N}(\mu, \sigma^2/n_i), \quad, i=1,2\dots,m
$$
where some of the (known) $n_i$'s are equal to 1. 
Write the likelihood function, take logarithms and you find the loglikelihood function as 
$$
\ell = \sum_{i=1}^m \cdot \log \sqrt{n_i} - \frac12 \log (2\pi\sigma^2) -\frac12 n_i \left(\frac{\bar{x}_i-\mu}{\sigma}\right)^2
$$
differentiating this with respect to $\mu$, setting equal to zero gives the likelihood equation for $\mu$, with solution
$$
\hat{\mu}=\frac{\sum n_i \bar{x}_i}{\sum n_i}
$$
that is, the solution is the weighted least squares estimator. 
To find an estimator for the variance $\sigma^2$, differentiate the loglikelihood function with respect to $\sigma^2$: First write
$$
S^2 = \sum n_i (\bar{x}_i -\hat{\mu})^2
$$
Then the concentrated loglikelihood can be written
$$
\ell^* = \frac12\sum \log(n_i) -\frac{m}{2}\log(2\pi \sigma^2) - \frac12\frac{S^2}{\sigma^2}
$$
differentiating, setting equal to zero and solving gives the estimator of the variance as
$$
\hat{\sigma^2} = \frac{S^2}{m}
$$
The same approach can be used if the parent distribution is something other than normal.
