1
$\begingroup$

I'm currently writing my Master's Thesis which focuses on building software to do some things.

I ran a few benchmarks for my program (Program A) and current alternatives (Program B) and want to present a CDF plot and a table with 50th, 99th, 99.9th up to 99.999th percentiles.

My problem is the following: Program A cannot have lower results due to the way it's designed, only slightly higher, but due to the variability between tests, it's very difficult to obtain the perfect test where the results for each percentile are all lower for Program B than for Program A.

So I want to combine several tests to reduce the variability, if possible, but I don't know how that is done. I don't have a strong statistics background.

My options currently are:

  • Calculate the percentiles for every test and average the values for equivalent percentiles across all tests - This is easy but I don't know if it's statistically relevant

  • Same as above but show the min, max and std deviation for each percentile - Might be better but I'd like to avoid this (if there is a better way) as it becomes harder to read.

  • Combine the raw data of each run and calculate the percentiles for all the data - I don't think this will reduce the amount of outliers.

  • Calculate CDFs for every run and somehow average the CDFs and display that - Basically the same problem I'm having now.

Could you suggest a better way to do this?

$\endgroup$
  • $\begingroup$ Grammar question: what does "I'm can't have" mean? Can you run paired tests and measure the difference? $\endgroup$ – EngrStudent - Reinstate Monica Oct 15 '19 at 12:52
  • 1
    $\begingroup$ "Combine the raw data of each run and calculate the percentiles for all the data" is the only accurate way to get the percentiles of all the data if you want them. What that implies about outliers is hard to predict: however, it is entirely possible that outliers won't seem so extraordinary when you add more data. $\endgroup$ – Nick Cox Oct 15 '19 at 14:25
0
$\begingroup$

It seems problematic that one program can never have lower results. Doesn't this invalidate the premise of parametric statistics? Notwithstanding that, cdf is based on a known mean and standard deviation, so a percentile does not have a standard deviation per se. Also, percentiles are not equally spaced along the distribution. There's a nice article in Wikipedia with a plot showing this. All this being said, you could ignore all this and treat a cdf as a quasi "score". This is often done in less scientific disciplines when converting a score from its original scale to something like 0 to 100. That may not be the best approach in this case.

$\endgroup$
  • 1
    $\begingroup$ I don't follow all of this. If I am estimating e.g. the 5th percentile from a sample there is uncertainty about that estimation and standard errors and confidence intervals can be defined. That is not the OP's issue, but in that sense it's not correct to say that a percentile does not have a standard deviation. $\endgroup$ – Nick Cox Oct 15 '19 at 14:39
  • $\begingroup$ It seems I misread what the op wanted. It seems that they don't want to average the actual percentile, but the value at the given percentile. In which case, I don't see a problem $\endgroup$ – logisticregress Oct 15 '19 at 14:45
  • $\begingroup$ I agree with Nick, option 3 is the statistically valid way to achieve this, in my opinion $\endgroup$ – logisticregress Oct 15 '19 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.