1
$\begingroup$

I'm told that the $Var(\overline{x}) \approx Var(X)/N$, where $\overline{x}$ is the mean of $N$ elements from $X$. However, I have a data set where these variances seem to differ by a factor of about 30.

A series of Bernoulli trials is run. The input data consists of numbers of successes and attempts made in an hour. I have some $400000$ trials unevenly distributed in around $1000$ hours. I take $M$ random samples of hours, and calculate their means as sum of successes divided by sum of attempts. I expect that if $E$ is the mean of these $M$ means, then their variance will be $E(1-E)/\text{sample size}$, however the real variance is found to be much bigger.

I suspect this effect in large part due to the uneven distribution of trials in hours and the way I sample them. Is that true? If so, can I correct for it? I tried aggregating hours with few events, but it didn't seem to help. Could something else be causing it? Should I expect the real variance to be correct if I use all the data rather than random samples?

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ Your first sentence is true only if the $X_i$'s are independent. $\endgroup$ – StatsStudent Feb 25 '15 at 7:36
  • $\begingroup$ Can you clarify the part about hours and uneven distribution. Do your samples have different numbers of trials? If so, how do you obtain $N$ for your formula? $\endgroup$ – Juho Kokkala Feb 25 '15 at 7:59
  • $\begingroup$ @JuhoKokkala, there could be anywhere from 0 to 20000 trials per hour though most have under 5000. The sample sizes are fairly normally distributed with mean of 100000 and standard deviation of 20000. I take N to be the mean of sample sizes. In this particular sampling the samples do overlap, though I've tried sampling without overlap too without much difference in the ratio of variances. $\endgroup$ – Karolis Juodelė Feb 25 '15 at 9:20
  • $\begingroup$ Is each individual Bernoulli trial characterized by the same probability of success? $\endgroup$ – Alecos Papadopoulos Feb 25 '15 at 10:17
  • 1
    $\begingroup$ This is called overdispersion, search this site. There can be many reasons. We really need much more details and context of your cae. $\endgroup$ – kjetil b halvorsen Aug 2 '17 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.