# Is there a qualitatively useful statistic or approach to an ill-behaving sample average?

I was running some timing simulations on a computer. As is often done with low latency things like this, I was running the relevant block in a loop $k$ times and then recording the total time over $k$ (average time). I repeated this thing $100$ times for $k=10000$ and $k=100000$ respectively and computed a mean and standard deviation for both sets of $100$ data points (so a sample mean of means and it's standard deviation). I noticed that this sample standard error was actually larger for the case of $k=100000$ than it was for the case of $k=10000$. I found this very perplexing as I am accustomed to thinking the sample mean estimator is distributed as $\sigma/ \sqrt{n}$. I therefore took this to mean that the draws (i.e. every run of the relevant block) were not i.i.d. Thinking about this I reasoned that varying system loads on a computer as well as changes in caches due to context switching and what not could mean the draws are most likely not identically distributed. I thought of using a runs test as well to test for independence but I don't have enough time measuring resolution to accurately observe individual iterations although I believe independence is less of an issue than the identically distributed assumption.

In any case, given this problem, I am not sure how to proceed from a statistical point of view when trying to provide a latency metric of qualitative value. (There seems to be no point to a sample mean if there is no population mean right?)

Perhaps this is an issue where only relative performance may be useful? (i.e. A is on average twice as fast as B but nothing can really be said about B because it's average latency is not well defined due to lack of environment control) Or is there some statistical tool (estimator, test, or adjustment) that can be made in a situation like this?

-
Did you keep all 100 sample averages for each of the two runs? Have you plotted them? Are there outliers? Are the data long-tailed? Do you have access to the internal code that you are trying to time? Is there a case in the code in which it could get lost in a loop for a long time? –  cardinal Sep 20 '11 at 13:55
To what do "A" and "B" refer in the last paragraph? –  whuber Sep 20 '11 at 15:48
A: I find an average time over 10,000 as a sample size and denoting that sample average by $t_1$..I then do this one hundred times: $t_1, t_2,\ldots,t_100$. These are all estimates of the mean and should have std of $\sigma/\sqrt{n}$ where $n = 10,000$ (if each timed iteration iid). B: I compute an average time over $100,000$ iterations now, and call it $z_1$..I repeat this $100$ times: $z_1, \ldots, z_100$ and have $100$ estimates of the mean which should now have std of $\sigma / \sqrt{n}$ for $n = 100,000$. What I see, however, is that this latter standard error is greater than the first.. –  Palace Chan Sep 20 '11 at 12:59