How can I statistically determine if my data on code run time measurements is “good”?

Question

I took some measurements of some data (code run time measurements, for those curious) of which I have no idea what the expected value is.

The data is discrete, and I have no idea what type of properties it has or distribution it follows.

The only thing that is known is that the values are more or less independent of each other. I say more or less, because there are some cache effects that can cause correlate one measurement with another, but I don't know how it may affect the measurements.

In addition, I do know that each measurement has a certain amount of granularity, since I can only measure to something like $\frac{1}{Frequency}$ accuracy, limited by the speed of my processor.

Given that I have $N$ total samples of data, what sorts of things can I do to say, for example, "I have $X$% confidence that the expected run time of this code should be $Y$ "?

I essentially want to determine how accurate my measurements are and to what degree I'm confident they are. I have no idea how to do this, as I've never had a chance to take a proper statistics class.

Answer 1

I'm focussing on your second last para and the fact that you haven't taken a stats class, and assuming you're mainly interested in saying something about the average runtime of your code?

The following is the simplest possible approach - I suspect that this is something you already know but it's not entirely clear to me from your post whether that's the case or not:

Run the code $N$ times and average the runtimes over those $N$ runs.

The Central Limit Theorem then implies that the sample mean you calculated came from a normal distribution (that is, unless the distribution of runtimes is rather strange then the sample means of collections of runtimes will be approximately normally distributed). From there you can use standard confidence bounds on normally distributed data to say the sort of things you'd like to about the runtime (example).