5
$\begingroup$

I have dataset where I am only seeing the aggregated data. Assume that there is an underlying random variable, $p$. The sample data is of the form $P_i = \sum^{n_i}_{j=1} p_{ij}$ with $i=1..N$. The individual $n_i$ are randomly distributed. I would like to estimate the variance of $p$ from this dataset.

In my case, about one third of the data has $n_i = 1$, so a simple approach is to use only this subset of the data, and estimate the sample variance of $p$ directly from it. However, this is discarding two thirds of the data. Is there a better method of calculating the sample variance using all of the data?

$\endgroup$
2
  • $\begingroup$ Unfortunately I do not have the sample variance of the groups. $\endgroup$
    – Obromios
    Oct 20, 2015 at 10:23
  • 1
    $\begingroup$ I am not so sure that you cannot recover the variance. My example above of choosing all groups with $n_i =1$ does provide a method, although only for data sets that have samples with $n_i = 1$. Extending this idea further, one approach would be to divide all the samples into groups with the same value of $n_i$. You can calculate the sample variance of each group. The shape of the graph of sample variance versus $n_i$ must have some information about the underlying variance. $\endgroup$
    – Obromios
    Oct 20, 2015 at 10:30

1 Answer 1

6
$\begingroup$

We can solve this with likelihood methods if we, in addition, can assume the original individual observations are iid normal. So suppose you have $$ x_{i,1},x_{i,2},\dots, x_{i,n_i} \sim \text{N}(\mu, \sigma^2) $$ independently, but you do only know $$ \bar{x}_i \sim \text{N}(\mu, \sigma^2/n_i), \quad, i=1,2\dots,m $$ where some of the (known) $n_i$'s are equal to 1.

Write the likelihood function, take logarithms and you find the loglikelihood function as $$ \ell = \sum_{i=1}^m \cdot \log \sqrt{n_i} - \frac12 \log (2\pi\sigma^2) -\frac12 n_i \left(\frac{\bar{x}_i-\mu}{\sigma}\right)^2 $$ differentiating this with respect to $\mu$, setting equal to zero gives the likelihood equation for $\mu$, with solution $$ \hat{\mu}=\frac{\sum n_i \bar{x}_i}{\sum n_i} $$ that is, the solution is the weighted least squares estimator.

To find an estimator for the variance $\sigma^2$, differentiate the loglikelihood function with respect to $\sigma^2$: First write $$ S^2 = \sum n_i (\bar{x}_i -\hat{\mu})^2 $$ Then the concentrated loglikelihood can be written $$ \ell^* = \frac12\sum \log(n_i) -\frac{m}{2}\log(2\pi \sigma^2) - \frac12\frac{S^2}{\sigma^2} $$ differentiating, setting equal to zero and solving gives the estimator of the variance as $$ \hat{\sigma^2} = \frac{S^2}{m} $$ The same approach can be used if the parent distribution is something other than normal.

$\endgroup$
4
  • $\begingroup$ This certainly gives an elegant final answer. I have upvoted it, but am looking for a solution that does not assume a particular distribution. Or perhaps one that deduces the parent distribution from the aggregated data, then applies the maximum likelihood method to that calculated distribution. $\endgroup$
    – Obromios
    Oct 20, 2015 at 21:12
  • $\begingroup$ As long as you only want the standard deviation, the formulas above can be developed without reliance on the normal distribution. A more important assumption is independence! An interesting additional answer is how much you loose by estimating the variance in this way, compared to having the complete data. I will add something about that to the answer when I have time. $\endgroup$ Oct 21, 2015 at 16:31
  • $\begingroup$ I was assuming i.i.d. Are you saying the final answer is the same, irregardless of the actual distribution? That would be very neat. $\endgroup$
    – Obromios
    Oct 21, 2015 at 22:04
  • 1
    $\begingroup$ The "m" in the denominator suggests that this estimator is biased. Is there a way to redo the math so that it's unbiased as well as consistent? $\endgroup$
    – pandichef
    Jan 22, 2021 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.