Statistics for benchmarking code

Question

My application has a factory class which has modular elements registered to it. A run method is called on the factory class which subsequently executes the run methods on the modules.

I am using PEAR PHP's Profiler to measure the speed of execution for each module in microseconds. So far I have been storing the module runtimes and the deltas of each runtime in my database.

This has produced somewhat interesting statistics when I graph the runtime (I can see spikes, etc). However, I don't have much experience with statistics, what kind of analysis would prove valuable for looking at code performance?

My intention is to catch naively written modules before deployment.

Thanks!

Answer 1

My thoughts are you should probably be using at survival analysis. Typically, as the name suggests, this measures time until death, or time until something breaks. But there's no reason you can run a survival analysis on code performance - essentially, looking at time until a piece of code completes.

This would allow you to compare performance over covariates - say the data your code is running on, machine, slightly different algorithms, etc.