# How can I statistically determine if my data on code run time measurements is “good”?

I took some measurements of some data (code run time measurements, for those curious) of which I have no idea what the expected value is.

The data is discrete, and I have no idea what type of properties it has or distribution it follows.

The only thing that is known is that the values are more or less independent of each other. I say more or less, because there are some cache effects that can cause correlate one measurement with another, but I don't know how it may affect the measurements.

In addition, I do know that each measurement has a certain amount of granularity, since I can only measure to something like $\frac{1}{Frequency}$ accuracy, limited by the speed of my processor.

Given that I have $N$ total samples of data, what sorts of things can I do to say, for example, "I have $X$% confidence that the expected run time of this code should be $Y$ "?

I essentially want to determine how accurate my measurements are and to what degree I'm confident they are. I have no idea how to do this, as I've never had a chance to take a proper statistics class.

-
When it comes to knowing whether measurement is valid, there is a lot to be said for domain specific experience. This allows you to form expectations about what the data should look like, and thus, you become more attuned to knowing when the data is deviating from your expectations. You may wish to elaborate more on the details of your specific measurement domain as I imagine most answers regarding assessing validity of data would be contingent on the domain. –  Jeromy Anglim Jun 5 '11 at 7:20
You may also find Zed Shaw's introduction to basic statistics for programmers helpful: zedshaw.com/essays/programmer_stats.html –  scw Jun 5 '11 at 23:46
@scw Good link but definitely NSFW –  Bob Durrant Jun 6 '11 at 6:34
@Bob right, I should've mentioned that -- hopefully one of the moderators can include (NSFW) in my comment. –  scw Jun 6 '11 at 7:14
Run the code $N$ times and average the runtimes over those $N$ runs.