In your case, the time is really the total time of the individual operations; the average speed is the total time divided by the number of operations. If you have that info (I assume you do) then you will not need harmonic means.
Let's take a famous internet trivium involving harmonic means, courtesy of wikipedia:
For instance, if a vehicle travels a certain distance d outbound at a
speed x (e.g. 60 km/h) and returns the same distance at a speed y
(e.g. 20 km/h), then its average speed is the harmonic mean of x and y
(30 km/h) – not the arithmetic mean (40 km/h). The total travel time
is the same as if it had traveled the whole distance at that average
In that example, the distance is not specified. If you have the distance, then the average speed is obtained directly by total distance / total time (as per same wikipedia entry).
In your case, the speed expressed in operations per unit of time is total number of operations / total time. If the number of operations is constant for each test run, you can even skip that and use the total time as a proxy for performance.
From the mention of bootstrapping, I am assuming that each data point (or row in the data) is an operation, and that each test run requires the app to do a fixed number of operations. Every operation is associated with a duration, and you can sum the durations to get the total test time. I will also assume that you bootstrapped by drawing N operations by model, B times, creating a bunch of simulated test runs. In the case that the tests are done by letting the app do as many operations as it can during a fixed time, the principle will remain the same. The bootstrapping allows you to have the same number of operations per app per bootstrap sample, so you'll be fine.
I'll describe what I did in a similar project and try to apply it to your situation. To test for superiority of, say, application A that seems faster than B, you could just look at the proportion of bootstrap samples where model A was faster than B. That will be your bootstrapped p-value. The confidence intervals will typically be the quantiles 0.025 and 0.975. If you need a two-sided test, take a look at the answers for this question:
Computing p-value using bootstrap with R
Results would like something like this: The bootstrapped estimate of the execution time for model A (xxx ms) was faster than B (yyy ms), p = 0.012.
Here the p-value is literally the proportion of bootstrap samples/iterations/trials where the A time was smaller than the B time, so my fictive 0.012 means 12 bootstrap runs for B = 1000. xxx is the median of the A times through all bootstrap samples, yyy is the same for B times. You can add the confidence intervals for the means, even report the bootstrapped difference of means with CIs.
For the SD, you can use the same technique. You will then have a robustly estimated CoV by a simple CoV = SD / mean.
What I did in my case is I compared every model with the one what was seemingly the best, i.e. in your case that's probably one with the shortest bootstrapped mean execution time. This avoids to test everything against everything, and you want to know which is best, not have exhaustive comparisons.