(This is one of those cases where I'm sure the answer is lurking somewhere but I don't know the correct terminology to find it.)

Sometimes with complex simulations it is intuitively necessary to simulate a very large number of individuals in order to get any meaningful result, perhaps into millions or even billions of individuals. Often it can be impractical to perform a full statistical analysis of such huge samples.

Furthermore, in some cases, the individuals will produce discrete values - low integers, - for instance, a number of failures in a given time which for an individual will usually be zero and sometimes one or rarely more than one. Some statistical methods I suspect are not well suited to such discrete values, but a continuous average value becomes meaningful over a large number of individuals.

So, it is tempting to, within each distinct class being simulated, group the individuals of that class randomly into batches of some large number of individuals, and then treat the average value of each batch as a single observation for the purposes of the analysis.

Is this a valid approach? Are there any pitfalls, or situations where the analysis would need to be adjusted to account for the fact that each observation is not actually an individual?

In particular, how would you deal with, and report, n?

It stands to reason that, in general, n would have to be the number of batches in order for the math to work correctly. Are there any exceptions to this?

In particular, are there interactions between effect size and the choice of batch size? For instance, if you simulate 500 million individuals in each class, can you have greater confidence for a smaller effect size by doing the analysis on 50,000 batches of 10,000 versus doing the analysis on 500 batches of one million?

And when you disclose results, would it be conventional to state n as the number of batches the analysis was done on, or the number of individuals involved?

  • $\begingroup$ Can you write down you question please in form of a statistical formula. Currently, it sounds to me you want to do some form of regression analysis? Or do you want to compare means? So what are clustering and what do you want to show? $\endgroup$ – Tom Pape Feb 24 at 16:41
  • $\begingroup$ @TomPape I'm actually interested in how this works (or doesn't work) in general - i.e., regardless of a specific statistical formula or goal. However, in my present case I'm most interested in hypothesis testing for differences between two classes. For instance, perhaps half the simulated individuals are subjected to an intervention and half are not, and we want to know the effect size of the intervention. But that's just one example. $\endgroup$ – Kevin Feb 24 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.