Ok, let's try again. The context of the original question is given below, but perhaps it helps to focus on the statistical aspect to get an answer.

What I got is a number of measurements in unit t. Those measurements are done while varying a larger number of variables. The impact of these variables is unknown and needs to be determined by interpreting the measurements.

Since the measurements are in absolute numbers and variable e of the experiments is typically the main variable of interested, having only two values, I tend to calculate the ratio of the measurements based on e. Using the ratio enables me to compare the impact of the other variables on t.

Now the question is: how to calculate the ratio properly?

What I did first was to pair up measurements of t where all variables are equal, and then take a measurement with e1 and one with e2 to calculate the ration. However, that results in high variations, and the result is not very useful because of the noise.

The second approach was to sort all measurements based on t and then calculate e1/e2. That reduces the noise a lot, but my half-knowledge tells me that is not the best way and perhaps not even correct.

A colleague suggested to use the cartesian product instead of pairing random or sorted measurements. That sounds like it would account better for the random nature of two arbitrary measurements paired up for comparison. But, I am still whether that is correct, or whether there are approaches with a theoretical backing around in the community.

Any pointers to approaches of how to work with and how to produce such ratios in a correct way a very welcome.

Context and Use-case

To give a bit of the context, I am measuring the performance of virtual machines (VMs), or systems software in general, and usually want to compare different optimizations for performance problem. Performance is measured in absolute runtime for a number of benchmarks, and usually for a number of configurations of a VM variating over used number of CPU cores, different benchmark parameters, etc. To get reliable results, each configuration is measure like 100 times. Thus, I end up with quite a number of measurements for all kind of different parameters where I am usually interested in the speedup for all of them, comparing the VM with and the VM without a certain optimization.

What I currently do is to pick one specific series of measurements. Lets say the measurements for a VM with and without optimization (VM-norm/VM-opt) running benchmark A, on 1 core.

Since I want to compare the results of the different benchmarks and number of cores, I can not use absolute runtime, but need to normalize it somehow. Thus, I pair up the 100 measurements for benchmark A on 1 core for VM-norm with the corresponding 100 measurements of VM-opt to calculate the VM-opt/VM-norm ratios.

When I do that taking the measurements just in the order I got them, I obviously have quite a high variation in my 100 resulting VM-opt/VM-norm ratios. So, I thought, ok, let's assume the variation in my measurements come from non-deterministic effects and the same effects cause variation in the same way for VM-opt and VM-norm. So, naively, it should be ok to sort the measurements before pairing them up. And, as expected, that reduces the variation of course.

  • $\begingroup$ I'm not sure why the ratios are important. Are you trying to just determine if a change 'significantly' lowered system or real time? $\endgroup$ – Jonathan Lisic Sep 23 '11 at 0:15
  • $\begingroup$ Ok, without ratios: how do I compare the impact of an optimization on different benchmarks/configurations? $\endgroup$ – smarr Sep 23 '11 at 6:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.