Repeating an experiment - more valuable than sample size?

Question

The setup is that I'm trying to understand how a computer program works, so I'm capturing some numbers every time a function is called. For example, I might be capturing the number of branches taken and the number of branches incorrectly predicted. During the course of running the program, a particular function might get called twenty or thirty thousand times. I have a fair amount of control over how many times a given function is called.

My initial plan was to calculate a mean and standard deviation using those 20-30k data points as my sample. However, my (computer science) professor suggested that I needed to rerun the experiment several times in order to calculate a standard deviation. So I would run my script five or six times, calculating a mean each time. Then I would use those five or six values to calculate a standard deviation. That doesn't make a lot of sense to me - it seems to me that if I want to understand how a given function is behaving, I should treat the data from each function call as a data point, and that the professor's method more or less throws away a lot of data.
However, I'm thinking that I may be making an unwarranted assumption that one run of the program is like another. In this case, I guess that running and presenting both sets of numbers would be a good, as I would capture how the functions behave at each call, and also see if there is behavior difference across calls.

So getting around to the question, is my initial impulse to use each function call as my data set correct/better, or should I calculate the data both ways and present both numbers?

Answer 1

For every experiment, one has explicit or implicit assumptions about all the boundary conditions that should not influence the experiment's outcome. When you run your program, maybe time of day should have no effect, it should not matter who presses the keyboard key to start the program, same for CPU architecture etc. You assume these things are unimportant because you have a well-established theory about what goes on when a computer runs a program.

However, some of these assumptions could be wrong because the theory is incomplete, only partly valid, or because you missed a connection between an unimportant boundary condition and a known influence (think of the email only within 500 miles anecdote, also check its FAQ). Repeating your experiment allows you to vary at least some of the boundary conditions and verify empirically that they indeed have no effect.

Quite often, replications reveal that a specific combination of circumstances that were not considered important can influence experiment's a great deal, see e.g., this New Yorker: the truth wears off article.

Answer 2

I assume that you analyse a stochastic algorithm. Repeated runs of the program may have different sources of variability than repeated function evaluations within a single run.

An example: the program may initialise a random number generator with the same seed in every run, which will make the results of repeated runs identical, but the function evaluations pseudo-random.

Answer 3

GaBorgulya said it quite well (+1), but I'd like to present a more conceptual perspective too.

If GaBorgulya's assumptions are correct, you can treat the random seed as a blocking factor. Imagine that each run of the program is a person and that for each person you measure blood pressure over time.

You could increase your sample size by taking more measurements on each person (like looking at all those thousands of data points) or by looking at more persons (like running the program more times). The former will tell you about the error within persons, and the latter will tell you about error between persons.

Because you are running a computer program and your program can be repeated if the seed is saved, there is no error within the "person" (the particular run of the program); whatever output you get is the true, complete output for that particular seed. You would like to know how much noise is created by the random seed.

On the other hand, your ability to adjust how many times a particular function is called blurs my definitions of error within and error between. It may help to discuss in a bit more detail what those functions are doing.