I was running some timing simulations on a computer. As is often done with low latency things like this, I was running the relevant block in a loop $k$ times and then recording the total time over $k$ (average time). I repeated this thing $100$ times for $k=10000$ and $k=100000$ respectively and computed a mean and standard deviation for both sets of $100$ data points (so a sample mean of means and it's standard deviation). I noticed that this sample standard error was actually larger for the case of $k=100000$ than it was for the case of $k=10000$. I found this very perplexing as I am accustomed to thinking the sample mean estimator is distributed as $\sigma/ \sqrt{n}$. I therefore took this to mean that the draws (i.e. every run of the relevant block) were not i.i.d. Thinking about this I reasoned that varying system loads on a computer as well as changes in caches due to context switching and what not could mean the draws are most likely not identically distributed. I thought of using a runs test as well to test for independence but I don't have enough time measuring resolution to accurately observe individual iterations although I believe independence is less of an issue than the identically distributed assumption.
In any case, given this problem, I am not sure how to proceed from a statistical point of view when trying to provide a latency metric of qualitative value. (There seems to be no point to a sample mean if there is no population mean right?)
Perhaps this is an issue where only relative performance may be useful? (i.e. A is on average twice as fast as B but nothing can really be said about B because it's average latency is not well defined due to lack of environment control) Or is there some statistical tool (estimator, test, or adjustment) that can be made in a situation like this?
