I'm currently writing my Master's Thesis which focuses on building software to do some things.
I ran a few benchmarks for my program (Program A) and current alternatives (Program B) and want to present a CDF plot and a table with 50th, 99th, 99.9th up to 99.999th percentiles.
My problem is the following: Program A cannot have lower results due to the way it's designed, only slightly higher, but due to the variability between tests, it's very difficult to obtain the perfect test where the results for each percentile are all lower for Program B than for Program A.
So I want to combine several tests to reduce the variability, if possible, but I don't know how that is done. I don't have a strong statistics background.
My options currently are:
Calculate the percentiles for every test and average the values for equivalent percentiles across all tests - This is easy but I don't know if it's statistically relevant
Same as above but show the min, max and std deviation for each percentile - Might be better but I'd like to avoid this (if there is a better way) as it becomes harder to read.
Combine the raw data of each run and calculate the percentiles for all the data - I don't think this will reduce the amount of outliers.
Calculate CDFs for every run and somehow average the CDFs and display that - Basically the same problem I'm having now.
Could you suggest a better way to do this?