How to calculate sample size for simulation in order to assert some level of goodness in my results?

Question

I am a stats newbie, so apologies in advance if I'm asking a braindead question. I have searched for answers to my question, but I find that many of the topics are either too specific, or quickly go beyond what I currently comprehend.

I have some simulation work that includes large datasets which become infeasible to simulate exhaustively. For the smallest of my datasets, an exhaustive run presents the following distribution of results from a total of 9180900 tests.

Result/Frequency:

• 0 7183804
• 1 1887089
• 2 105296
• 3 4571
• 4 140

What the numbers mean does not matter; what matters is that the larger datasets I have can stretch into billions of tests, and become far too time consuming to run. I need to constrain the workload.

I feel I ought to be able to sample from the full set of tests to derive a distribution for the sample, and infer (within some bounds) that the results of an exhaustive simulation would exhibit roughly the same distribution. There is no bias inherent in the tests which are run, so uniformly randomly choosing inputs ought to provide a valid sample.

What I do not yet understand is how I should go about selecting my sample size. In particular, the distribution exhibits a strange tail, and I fear that sampling too small will lose the lower frequencies. (The 140 occurrences of '4' account for only 0.0015% of the population!)

So, my question is, what is the best way of calculating a sample size with which I can assert some level of goodness in my results?

Or, am I asking the wrong question?

Answer 1

I think the answer to your question is a couple other questions: how rare does a given test outcome need to be before you don't care about it? How certain do you want to be that you'll actually find at least test that comes out that way if it occurs right at the threshold where you've stopped caring about it. Given those values you can do a power analysis. I'm not 100% confident whether you need to do a multinomial (involving more than one outcome) power analysis or not, I'm guessing a binomial one (either the rare test or not) will work just fine, e.g. http://statpages.org/proppowr.html. Alpha = .05, Power = 80%, Group on proportion 0, Group 1 proportion .0015. Relative sample size, 1; total - just south of 13,000 tests. At which the expected number of test 4s is ~20.

That will help you find the number of tests you need to have to detect one of those rare occurring results. However if what you really care about is relative frequency, the problem is harder. I'd conjecture that if you simply multiplied the resulting N from the power analysis by 20 or 30 you'd find a reasonable guess.

In practice, if you don't really need to decide the number of tests ahead of time, you might consider running tests until you get 20 or 30 result 4s. By the time you've gotten that many 4s you should start to have a reasonable though not absolute estimate of their relative frequency IMO.

Ultimately - there are trade-offs between number of tests run and accuracy. You need to know how precise you want your estimates to be before you can really determine how many is "enough".

Answer 2

I think that power analysis is too elaborate for what you're trying to do, and might let your down.

With a sample size north of 9 million, I think your estimate for p = Pr(X > 3) = 0.000015 is pretty accurate. So you can use that in a simple binomial(n, p) model to estimate a sample size.

Let's say your goal is to observed at least one "Large" event with a probability of 99.9%. Then Pr(L > 0) = 1 - Pr(L = 0) = 1 - 0.999985^n = 0.999 and your desired sample size is n = ln(0.001)/ln(0.999985) = 460514.

Of course, if you're feeling lucky and are willing to take a 10% chance of missing a Large event, you only need a sample size of n = 153505. Tripling the sample size cuts your chance of missing the Large event by a factor of 100, so I'd go for the 460,000.

BUT...if you're looking for FIVE's, their probability is just south of 1/9180902 and to observe at least one of THOSE with 99.9% probability, you'd need a sample size of about 63.4 million!

Do heed DrKNexus' advice about updating your estimate of the probabilities for the Large events, since it might not be constant across all your datasets.