# Is there a qualitatively useful statistic or approach to an ill-behaving sample average?

I was running some timing simulations on a computer. As is often done with low latency things like this, I was running the relevant block in a loop $k$ times and then recording the total time over $k$ (average time). I repeated this thing $100$ times for $k=10000$ and $k=100000$ respectively and computed a mean and standard deviation for both sets of $100$ data points (so a sample mean of means and it's standard deviation). I noticed that this sample standard error was actually larger for the case of $k=100000$ than it was for the case of $k=10000$. I found this very perplexing as I am accustomed to thinking the sample mean estimator is distributed as $\sigma/ \sqrt{n}$. I therefore took this to mean that the draws (i.e. every run of the relevant block) were not i.i.d. Thinking about this I reasoned that varying system loads on a computer as well as changes in caches due to context switching and what not could mean the draws are most likely not identically distributed. I thought of using a runs test as well to test for independence but I don't have enough time measuring resolution to accurately observe individual iterations although I believe independence is less of an issue than the identically distributed assumption.

In any case, given this problem, I am not sure how to proceed from a statistical point of view when trying to provide a latency metric of qualitative value. (There seems to be no point to a sample mean if there is no population mean right?)

Perhaps this is an issue where only relative performance may be useful? (i.e. A is on average twice as fast as B but nothing can really be said about B because it's average latency is not well defined due to lack of environment control) Or is there some statistical tool (estimator, test, or adjustment) that can be made in a situation like this?

-
Did you keep all 100 sample averages for each of the two runs? Have you plotted them? Are there outliers? Are the data long-tailed? Do you have access to the internal code that you are trying to time? Is there a case in the code in which it could get lost in a loop for a long time? – cardinal Sep 20 '11 at 13:55
To what do "A" and "B" refer in the last paragraph? – whuber Sep 20 '11 at 15:48

 A: I find an average time over 10,000 as a sample size and denoting that sample average by $t_1$..I then do this one hundred times: $t_1, t_2,\ldots,t_100$. These are all estimates of the mean and should have std of $\sigma/\sqrt{n}$ where $n = 10,000$ (if each timed iteration iid). B: I compute an average time over $100,000$ iterations now, and call it $z_1$..I repeat this $100$ times: $z_1, \ldots, z_100$ and have $100$ estimates of the mean which should now have std of $\sigma / \sqrt{n}$ for $n = 100,000$. What I see, however, is that this latter standard error is greater than the first.. – Palace Chan Sep 20 '11 at 12:59 @PalaceChan I think Adam is right, you really have a sample size of 100 and should look at it that way. Doing something 100000 times and recording the total elapsed time is not quite the same as timing something 100000 times and taking the mean. A lot of things could be happening that account for the larger standard error (say garbage collection because the larger loop exhausted available memory) and you should be trying to understand that rather than looking for a statistical solution. – Gaël Laurans Mar 18 '12 at 11:28