# How to Average Percentiles

I'm currently writing my Master's Thesis which focuses on building software to do some things.

I ran a few benchmarks for my program (Program A) and current alternatives (Program B) and want to present a CDF plot and a table with 50th, 99th, 99.9th up to 99.999th percentiles.

My problem is the following: Program A cannot have lower results due to the way it's designed, only slightly higher, but due to the variability between tests, it's very difficult to obtain the perfect test where the results for each percentile are all lower for Program B than for Program A.

So I want to combine several tests to reduce the variability, if possible, but I don't know how that is done. I don't have a strong statistics background.

My options currently are:

• Calculate the percentiles for every test and average the values for equivalent percentiles across all tests - This is easy but I don't know if it's statistically relevant

• Same as above but show the min, max and std deviation for each percentile - Might be better but I'd like to avoid this (if there is a better way) as it becomes harder to read.

• Combine the raw data of each run and calculate the percentiles for all the data - I don't think this will reduce the amount of outliers.

• Calculate CDFs for every run and somehow average the CDFs and display that - Basically the same problem I'm having now.

Could you suggest a better way to do this?

• Grammar question: what does "I'm can't have" mean? Can you run paired tests and measure the difference? – EngrStudent - Reinstate Monica Oct 15 '19 at 12:52
• "Combine the raw data of each run and calculate the percentiles for all the data" is the only accurate way to get the percentiles of all the data if you want them. What that implies about outliers is hard to predict: however, it is entirely possible that outliers won't seem so extraordinary when you add more data. – Nick Cox Oct 15 '19 at 14:25