How to measure the consistency of improvement on different conditions?

Question

I want to measure whether the speed improvement of method 1 over method 2 is consistent on different conditions. Below are two examples of the speedup values of method 1 over method 2 on 5 conditions.

case 1:

case 2:

I'll say that in case 1, the speedup is consistent, while for case 2, the speedup is inconsistent.

However, I find it hard to decide whether the speedup is consistent for case 3:

case 3:

I don't want my conclusions sounds subjective. So, is there any statistical tests to mathematically measure such consistency?

Answer 1

I think one of the best methods is anomaly detection. However, it requires for adding a time-series variable instead of your variable condition (condition1, condition2, etc.). There is an amazing package written in R programming language called anomalize. First, I provide the results based on the IQR (i.e., interquartile range) method: $$IQR = Q3 - Q1 = P75\% - P25\%$$

   case   speed anomaly
   <chr>  <dbl> <chr>  
 1 case 1   1.2 No     
 2 case 1   1.3 No     
 3 case 1   1.1 No     
 4 case 1   1.1 No     
 5 case 1   1.5 No     
 6 case 2   1.2 No     
 7 case 2   1.1 No     
 8 case 2  20   Yes    
 9 case 2  30   Yes    
10 case 2 100   Yes    
11 case 3   1.2 No     
12 case 3   1.1 No     
13 case 3   2   No     
14 case 3   3   No     
15 case 3   4   No  

Only the second case at speedups 20, 30, and 100 showed anomaly.

The IQR Method uses an innerquartile range of 25 the median. With the default alpha = 0.05, the limits are established by expanding the 25/75 baseline by an IQR Factor of 3 (3X). The IQR Factor = 0.15 / alpha (hense 3X with alpha = 0.05). To increase the IQR Factor controling the limits, decrease the alpha, which makes it more difficult to be an outlier. Increase alpha to make it easier to be an outlier.

The other method, which is based on measures of relative dispersion (e.g., coefficient of variation, coefficient of quartile variation):
$$CV = \biggl(\frac{\sigma}{\mu}\biggr)\times100,$$ (Albatineh, et al 2014)
$$CQV = \biggl(\frac{Q_3-Q_1}{Q_3+Q_1}\biggr)\times100$$ (Altunkaynak and Gamgam, 2018)

Since cqv and cv are unitless, they are useful for comparison of variables with different units. They are also measures of homogeneity/consistency (Bonett, 2006) (Altunkaynak and Gamgam, 2018). These measures can be efficiently calculated with 95% confidence intervals by the recently released cvcqv R package (on CRAN). Because your example contains only 15 speedup values with a non-normal distribution, cqv is a better function to find out the amount of variability (i.e., dispersion):

CQV of case 1: 
        est lower upper                                         description
bonett  8.3    NA    NA                              cqv with Bonett 95% CI
norm    8.3   1.5  18.5                cqv with normal approximation 95% CI
basic   8.3   1.2  16.6                     cqv with basic bootstrap 95% CI
percent 8.3   0.0  15.4                cqv with bootstrap percentile 95% CI
bca     8.3   0.0  15.4 cqv with adjusted bootstrap percentile (BCa) 95% CI

CQV of case 2:
         est lower upper                                         description
bonett  92.3    NA    NA                              cqv with Bonett 95% CI
norm    92.3  48.9 191.5                cqv with normal approximation 95% CI
basic   92.3  86.8 184.6                     cqv with basic bootstrap 95% CI
percent 92.3   0.0  97.8                cqv with bootstrap percentile 95% CI
bca     92.3   4.3  97.8 cqv with adjusted bootstrap percentile (BCa) 95% CI

CQV of case 3:
         est lower upper                                         description
bonett  42.9    NA    NA                              cqv with Bonett 95% CI
norm    42.9  23.6  88.4                cqv with normal approximation 95% CI
basic   42.9  28.9  85.8                     cqv with basic bootstrap 95% CI
percent 42.9   0.0  56.9                cqv with bootstrap percentile 95% CI
bca     42.9   0.0  56.9 cqv with adjusted bootstrap percentile (BCa) 95% CI

As you can see, case 1 showed minimal dispersion/variation (~8%); case 2 showed severe variation (~92%), and case 3 showed moderate variation (~43%).