A performance test of a software application measures the maximum rate (operations per second), which this application can handle. The test is repeated multiple times each iteration yielding the determined maximum rate. So, we have a sample of rates to be analyzed statistically. It should be analyzed if the result is useful (maximum dispersion as per CoV, minimum confidence as per the size of the bootstrapped confidence interval) and the results should be summarized in a single number. The size of the sample is rather small (<= 10).

  1. AFAIK the harmonic mean should be used to summarize such rates instead of the arithmetic mean or am I wrong?
  2. Do I also have to use variants of other statistical measures considering the fact, that the sample consists of rates, i.e.: Are specialized formulas for the standard deviation and the coefficient of variation different from the commonly used formulas needed? In the common examples for the SD and the CoV the arithmetic mean (/ expected value) is used. Would I have to replace the arithmetic mean in these cases? I found this question, why I think, that specialized approaches are need, but I have much too little knowledge about statistics to judge my specific case.

In your case, the time is really the total time of the individual operations; the average speed is the total time divided by the number of operations. If you have that info (I assume you do) then you will not need harmonic means.

Let's take a famous internet trivium involving harmonic means, courtesy of wikipedia:

For instance, if a vehicle travels a certain distance d outbound at a speed x (e.g. 60 km/h) and returns the same distance at a speed y (e.g. 20 km/h), then its average speed is the harmonic mean of x and y (30 km/h) – not the arithmetic mean (40 km/h). The total travel time is the same as if it had traveled the whole distance at that average speed.

In that example, the distance is not specified. If you have the distance, then the average speed is obtained directly by total distance / total time (as per same wikipedia entry).

In your case, the speed expressed in operations per unit of time is total number of operations / total time. If the number of operations is constant for each test run, you can even skip that and use the total time as a proxy for performance.

From the mention of bootstrapping, I am assuming that each data point (or row in the data) is an operation, and that each test run requires the app to do a fixed number of operations. Every operation is associated with a duration, and you can sum the durations to get the total test time. I will also assume that you bootstrapped by drawing N operations by model, B times, creating a bunch of simulated test runs. In the case that the tests are done by letting the app do as many operations as it can during a fixed time, the principle will remain the same. The bootstrapping allows you to have the same number of operations per app per bootstrap sample, so you'll be fine.

I'll describe what I did in a similar project and try to apply it to your situation. To test for superiority of, say, application A that seems faster than B, you could just look at the proportion of bootstrap samples where model A was faster than B. That will be your bootstrapped p-value. The confidence intervals will typically be the quantiles 0.025 and 0.975. If you need a two-sided test, take a look at the answers for this question: Computing p-value using bootstrap with R

Results would like something like this: The bootstrapped estimate of the execution time for model A (xxx ms) was faster than B (yyy ms), p = 0.012.

Here the p-value is literally the proportion of bootstrap samples/iterations/trials where the A time was smaller than the B time, so my fictive 0.012 means 12 bootstrap runs for B = 1000. xxx is the median of the A times through all bootstrap samples, yyy is the same for B times. You can add the confidence intervals for the means, even report the bootstrapped difference of means with CIs.

For the SD, you can use the same technique. You will then have a robustly estimated CoV by a simple CoV = SD / mean.

What I did in my case is I compared every model with the one what was seemingly the best, i.e. in your case that's probably one with the shortest bootstrapped mean execution time. This avoids to test everything against everything, and you want to know which is best, not have exhaustive comparisons.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.