2
$\begingroup$

Suppose I have numerical data describing the total process time for a given software simulation. This data is broken up into 5 groups (Base, AD1, AD2, AD3, AD4) each detailing a different performance intervention with approximately the same number of observations for each group.

My goal is to determine if the performance interventions result in significantly different alive times than the base case and to determine which intervention is "best". "Best" being defined as the least amount of process time.

To clarify, my data is comprised of all the "regression-tests" in our code framework. So at this point I am looking at a high-level what the interventions do to overall process time but eventually will create sub-categories within each intervention to determine inter-group effect on process time.

My data has some extreme outliers as can be seen from this graphic:

enter image description here

My hypothesis is as follows:

$$ H_{0}: \mu_{\text{base}} = \mu_{\text{AD1}} = \mu_{\text{AD2}} = \mu_{\text{AD3}} = \mu_{\text{AD4}} $$

$$ H_{A}: \text{Not all means equal} $$

I am unsure what my hypothesis would be in determining the best "metric". I am also unsure if using the mean is appropriate in this circumstance given the outliers in my data.

My idea is to use some form of ANOVA or Krukall Wallis test and then maybe a Tukey Test to determine which one is best? I am open to Bayesian or Frequentist approaches to this. I might be over thinking this as well.

EDIT: I have made some good progress using Martin's suggestions and have been following along a tutorial found at:

https://www.andrewheiss.com/blog/2019/01/29/diff-means-half-dozen-ways/

and

http://www.flutterbys.com.au/stats/tut/tut7.4b.html

Through this I've built a model that kind of approximates the density of my alive times but unsure if I can make it better or where to go from here:

fit1 <-brm(alive_time ~ intv, data = dat, iter = 10000, warmup = 1000,
    chains = 3, thin = 2, refresh = 0, cores = 12,
    family = student(link = 'log'))

pp_check(fit1, nsamples = 1e3) + xlim(0, 40000)

enter image description here At the request of @Ben I've went ahead and plotted the densities using library(ggridges) on a log10 scale. Here is code and resulting plot:

dat %>% 
    ggplot(aes(y = intv, x = log10(alive_time), height = ..density..)) +
    geom_density_ridges2(stat = 'density') +
    labs(x = "Alive Time", y = "Interventions", 
         title = "Standard Ridgeline Plot")

enter image description here

I've also provided my data using dput(dat) in R below:

dat <- structure(list(intv = structure(c(4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 
4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 4L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 
5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 5L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 3L, 
3L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L), .Label = c("Base", "Ad", "Timestep", "Daniel1", "Shane1"
), class = "factor"), alive_time = c(10477.486474353, 15462.898424906, 
7.665776406, 8.414731227, 9.210761743, 4.017424642, 4.493777922, 
4.918436628, 2.246632132, 2.580258666, 2.836145724, 1.540199249, 
1.721755586, 1.898067896, 1.206603031, 1.444920648, 1.586049433, 
1.223530677, 1.374306366, 1.514164025, 2139.804058848, 2268.37153924, 
2178.079627582, 14390.37095987, 3835.342706029, 22252.500414809, 
20465.873024157, 15316.155927163, 23934.43387019, 28684.205118383, 
14395.69150759, 10962.580961668, 12414.105887542, 18584.658076431, 
19239.442516556, 19098.132572659, 13617.944780817, 22924.620553863, 
8750.700240409, 7584.063561984, 13274.532326964, 15149.933944385, 
25756.285614756, 19176.849027582, 12681.561851204, 26455.587340254, 
4233.929780132, 2343.967515272, 3891.569022885, 3524.562912168, 
2295.689004345, 2728.340292475, 3523.230399729, 2898.215419231, 
932.85045787, 791.070993243, 983.207437087, 444.83004738, 444.33692306, 
613.183019205, 851.386419484, 2662.570880028, 23735.121412053, 
14735.013272352, 4752.198156618, 28081.623057182, 28015.177471506, 
27628.062910486, 14792.113743361, 15638.816858656, 11234.280333881, 
24979.333642328, 24946.090644972, 4351.134506585, 4041.531976453, 
831.871612316, 294.260439763, 184.862397718, 9581.000459489, 
4890.532827258, 2062.08722422, 1946.648436285, 2048.737383734, 
1894.671220269, 2049.824549245, 1874.892934304, 1958.000087582, 
1864.139659147, 2145.544504207, 2122.117251084, 2256.390330179, 
2128.925847905, 945.96989839, 833.88139152, 882.556095446, 817.669087374, 
881.799307963, 814.338140236, 911.836684402, 797.269973129, 856.967394152, 
785.393070854, 816.319586729, 741.970131347, 1405.752754121, 
1292.442324213, 882.920916393, 813.523062245, 852.821786426, 
797.822076345, 763.491906962, 694.07273145, 834.63872812, 752.779319785, 
891.583529842, 801.580374674, 1563.664771053, 1503.632590209, 
1337.548753216, 1219.890817284, 1481.882001271, 1383.941468317, 
1450.762971259, 1314.172618048, 782.438301925, 710.769508183, 
1715.480938156, 1621.877874192, 1729.383936174, 1700.499525546, 
920.002176376, 852.479239639, 1410.626143679, 1420.261666573, 
1167.860063204, 1072.899506234, 1041.777046435, 950.473338333, 
683.850159879, 607.631964906, 721.172564459, 651.380077874, 187.309558421, 
171.39616093, 225.945850631, 206.626979269, 1728.408443902, 1570.131804674, 
1204.724222054, 1123.12347573, 1090.879121324, 1018.472985765, 
1023.159600098, 1086.03069888, 845.186872167, 789.796143947, 
960.961935916, 889.133671775, 11620.846130882, 46439.200037553, 
7259.855787583, 2653.115423044, 3199.239825203, 3440.436345477, 
673.920290883, 2982.450748097, 3315.823293488, 4486.025600435, 
167.051735438, 4512.417969168, 3566.115598728, 3241.520018281, 
5439.228304917, 4530.621770649, 2270.911732484, 5633.447199621, 
5112.40159827, 4935.447188634, 2460.384279123, 4964.3893024, 
5066.702498401, 2439.11978179, 3168.333363353, 1349.129484372, 
3500.312713384, 3918.150521527, 2726.80697369, 3400.288018976, 
2992.987326537, 978.367936371, 7703.767023843, 9190.693880286, 
9468.732323055, 11282.88155395, 891.958311166, 987.139602593, 
3513.805206819, 3528.026260471, 4082.680791945, 3089.133899853, 
3561.988491577, 3739.890216091, 4372.556410666, 13107.744535865, 
9714.380801656, 307.136814211, 14897.446630047, 267.651155027, 
11362.784352757, 14984.23798044, 11191.877833735, 15342.983141659, 
7.828128698, 8.623044825, 9.402143513, 4.112654521, 4.618458667, 
5.013303361, 2.275603921, 2.648975134, 2.87794921, 1.572978061, 
1.77455654, 1.933662805, 1.23745918, 1.504298648, 1.612627601, 
1.253439973, 1.407773352, 1.539620222, 2214.819307841, 2398.545663917, 
2299.12900277, 13281.1519391, 3887.276134524, 21394.55865882, 
20692.443567153, 15567.354417516, 23061.46634962, 27790.784615372, 
14427.931217373, 10952.574267423, 12386.612301246, 18428.919087867, 
19610.967793897, 18850.79790296, 13528.941636007, 22930.097302316, 
8716.669216436, 7588.965039386, 13385.207299096, 15201.187886311, 
24922.304407588, 19328.580950517, 12711.375623171, 26416.665976643, 
14541.598161982, 2342.731684273, 3863.917400016, 3278.050308891, 
2176.345707484, 2789.556594967, 3515.14764991, 2849.670591005, 
988.170042563, 801.335176311, 975.49276559, 446.501566006, 447.048951164, 
637.708437447, 846.770587468, 2508.649063334, 23819.336609558, 
13825.465744509, 4733.791002775, 28190.791238796, 27300.683772099, 
27748.596178692, 14721.736257948, 15322.369640528, 11040.193308981, 
39990.377979793, 24404.385388334, 4350.540240463, 4056.009936785, 
931.116539626, 319.765335272, 223.525010321, 2697.183786116, 
7.984119199, 2073.0474675, 9550.203165969, 4972.404636066, 2319.29631474, 
2021.070429394, 2275.218714612, 2124.529387947, 2154.100666089, 
1940.940076053, 2099.558783077, 1922.261148239, 2086.575112868, 
1889.169712931, 2454.566237798, 2192.448736569, 921.950686796, 
868.941175468, 929.183215374, 844.62921356, 1052.221090634, 964.676508192, 
972.585176828, 927.06502814, 986.370431404, 846.000516324, 848.91101562, 
782.230145341, 1654.979174586, 1395.444206727, 922.785503687, 
870.224164995, 1027.069093636, 869.575080871, 800.241833112, 
763.594327035, 909.588004114, 829.168474781, 1009.333139029, 
903.217640648, 1686.586421196, 1553.570950707, 1806.876927074, 
1506.493537691, 1814.172930367, 1657.788556037, 1795.147647595, 
1632.632665616, 1001.236477721, 956.666780932, 1988.905741483, 
1687.30776166, 2006.917264338, 1710.977749095, 944.444070572, 
865.737528435, 1602.364564823, 1375.797141158, 1324.864043672, 
1237.46497257, 1058.142749297, 950.418634537, 705.339442371, 
648.197734393, 856.468516393, 753.644844145, 193.146793088, 176.876608114, 
243.2907308, 226.483472146, 1801.302295661, 1671.775704224, 1123.592254691, 
1018.185336185, 1015.661905041, 920.406374837, 1054.103594745, 
1117.821032257, 1016.319716921, 947.71715952, 1018.900292434, 
953.394801622, 7410.821667027, 3384.885699843, 3472.247453662, 
4176.430670791, 636.668399206, 3097.944262372, 3152.803280179, 
4615.683510109, 158.730013218, 4235.470707974, 3398.953531263, 
3167.87590064, 5186.787018277, 4447.984331733, 2266.629095619, 
7543.009550159, 5347.469612856, 4629.582045525, 1915.129156228, 
4839.399452395, 4910.039662286, 2403.711364161, 3142.290949401, 
1461.608181538, 3462.281670435, 3871.013786235, 2720.766241738, 
3536.153746303, 2980.297139849, 996.217318177, 7723.439677371, 
9205.63009471, 9829.557310343, 11711.878150449, 922.727845076, 
1000.441968311, 3483.646262623, 3451.435345587, 3966.732580686, 
3089.466852748, 3572.152153, 3769.40785802, 4413.80870026, 13077.689399977, 
9904.679566789, 307.974384182, 12868.184606796, 265.701931691, 
11645.575544979, 14909.856861772, 11481.941179964, 16511.094518736, 
6.355724876, 7.029256677, 7.661710719, 3.328646758, 3.76357336, 
4.088905015, 1.875240383, 2.144813038, 2.356469994, 1.276509639, 
1.443256088, 1.577887777, 0.999972174, 1.258132352, 1.32229044, 
1.014554394, 1.144693429, 1.250646139, 1819.331183662, 1973.204714997, 
2241.829591422, 13571.668422278, 3815.354863454, 21478.599747204, 
20337.568289176, 15218.206957769, 22788.977537733, 27496.692554082, 
14759.939253915, 11020.324349908, 12016.256765476, 18433.396609096, 
19679.896331324, 18703.351836913, 13388.375769699, 22842.379256657, 
8661.979948624, 7765.50187174, 11040.604890491, 12324.42653671, 
24960.857193829, 19085.02325561, 12635.964667845, 26374.220175371, 
4177.988625576, 2329.02354921, 4114.925857035, 3470.389881276, 
2155.973841941, 2706.623991752, 3530.035649796, 2849.919815446, 
939.853172928, 836.935501803, 977.527673498, 453.896154977, 440.09072385, 
606.073291705, 840.236903446, 2520.632528438, 23814.890833368, 
14209.168122833, 4833.791239664, 27763.945063929, 26948.81390054, 
27072.016742267, 14507.132775206, 15266.768641656, 11079.145330692, 
24090.4075872, 24411.349698724, 4297.171179291, 4021.557876917, 
829.933891223, 299.054874119, 200.703598451, 2613.501849335, 
8.2063636, 2029.341459918, 9685.923795909, 4873.010096164, 2028.10133965, 
1834.213433725, 2096.966084336, 1827.424802494, 2026.292667234, 
1807.169694145, 2154.672196204, 1850.272501026, 2067.327058762, 
1878.331686777, 2257.275781779, 2013.144377271, 911.402626999, 
818.80063028, 859.364166903, 798.611057489, 890.80310288, 798.983163258, 
865.74126983, 794.560467193, 871.791071736, 775.32144092, 848.841401229, 
767.576850803, 1516.637728836, 1351.12450758, 881.522522404, 
807.124965971, 824.421898511, 748.185282956, 769.732554569, 708.593405995, 
825.390729975, 742.073176756, 875.532132541, 789.678430308, 1540.223702715, 
1365.121657245, 1409.524503707, 1237.051842982, 1465.958294012, 
1333.533910288, 1628.998928184, 1512.026563742, 970.513907986, 
880.828074816, 1628.627129081, 1476.793427861, 1667.83870288, 
1523.150336547, 1005.54297205, 914.605599319, 1374.878915707, 
1225.263090486, 1200.116130938, 1065.922882089, 1074.935257804, 
983.372293322, 713.507423357, 653.060598564, 743.436312856, 675.088117297, 
188.355701823, 168.644767707, 226.568184331, 205.413182984, 1679.764431886, 
1537.103338095, 983.293671476, 905.747310279, 1075.808216583, 
989.106508957, 1017.762151578, 1070.578732749, 1009.057437336, 
900.449059368, 978.223214616, 897.107128479, 6889.536054983, 
2708.03069678, 3031.437100759, 3560.933318343, 656.379732423, 
2939.150029034, 3229.716465672, 4600.28301051, 171.035837069, 
4424.623051697, 3536.763574135, 3147.11260987, 5296.576268739, 
4419.560451017, 2284.932598553, 5393.891198889, 4980.214202219, 
4532.147314084, 2636.166341455, 4925.928053733, 4922.492636589, 
2470.84062464, 3151.305846093, 1339.746317104, 3443.120865724, 
3861.378318655, 2710.572297575, 3421.031231996, 3045.44437603, 
977.70708094, 7768.828592507, 9133.812637472, 9590.997249674, 
11236.764837554, 883.913459746, 990.606527643, 3421.66280523, 
3591.735482467, 4017.343033315, 3081.21936964, 3543.223608265, 
3692.470168507, 4326.174327684, 12747.697743626, 10102.625037673, 
350.837173632, 15024.643932649, 261.14886504, 11584.525414532, 
13924.736299545, 12857.687272456, 17048.110955675, 6.246834566, 
6.881466688, 7.421381818, 3.290735747, 3.673799017, 3.986093252, 
1.836880951, 2.096229927, 2.291800417, 1.255165968, 1.502052612, 
1.53410418, 0.985414042, 1.181204468, 1.28441096, 1.00113897, 
1.121030269, 1.225056385, 2210.130444413, 2377.642025226, 2256.375947833, 
13489.794505582, 3754.142582694, 21359.28489732, 20616.190794592, 
15957.898044331, 23100.773410943, 27495.127084992, 14212.197542456, 
11013.389693721, 11980.292852242, 18348.721758173, 19287.691571951, 
18729.031879416, 13587.106790038, 23543.20943874, 9026.242565173, 
7582.680182644, 13323.018237341, 15146.220617202, 24654.699328121, 
19196.542443184, 12656.878361669, 26101.823403022, 4144.774302329, 
2426.864544501, 4093.087235011, 3433.989631948, 2176.709850973, 
2791.287290145, 3509.363364461, 2887.728488813, 938.685786866, 
808.452451307, 978.125592565, 456.852526414, 443.706321084, 608.390902791, 
853.033446964, 2413.235253012, 23633.218575305, 14122.9963913, 
4762.085998406, 39970.160116301, 27606.297084149, 27212.392648057, 
14768.077456856, 15241.474300417, 11317.084417127, 24567.185970313, 
24337.610854767, 4328.518219795, 4042.39507137, 803.965582122, 
270.282972987, 168.3359665, 2565.451984753, 7.944202868, 2020.416784042, 
9642.870608299, 4846.97771672, 2032.32574534, 1893.23003522, 
1956.06802972, 1822.093055208, 2021.301509325, 1886.582841852, 
2211.147743323, 1966.258590691, 2136.344391969, 1961.278791282, 
2306.280002658, 2172.7889755, 902.546391746, 825.741462499, 887.723997045, 
804.549397232, 882.402084914, 807.16433189, 880.236010728, 807.606717052, 
835.336453687, 762.202091052, 790.726376406, 714.647061412, 1341.657971345, 
1234.358383205, 903.73827845, 823.957750278, 976.280369611, 883.887712592, 
756.452580964, 692.360533558, 837.671345559, 764.448153533, 857.709211118, 
766.281874923, 1513.754878524, 1388.000615287, 1381.850261077, 
1256.669585298, 1466.377393976, 1345.110011771, 1645.592607403, 
1518.797804613, 888.401852326, 825.262540608, 1673.959155636, 
1507.622793317, 1775.982473917, 1611.813054698, 931.445425579, 
866.355945677, 1355.16176587, 1237.616800637, 1197.913452648, 
1097.634224865, 1001.851062876, 908.587257338, 664.559701168, 
592.474707, 697.80596222, 643.054630344, 183.018618446, 168.162314471, 
223.295369939, 212.446657568, 1724.085304572, 1606.955086898, 
952.48937542, 887.638743056, 1068.245320642, 989.875405855, 986.605130777, 
1053.72521362, 975.819999156, 900.386792357, 970.109869966, 898.89271337, 
10781.208046887, 45037.942550129, 7245.167103019, 2712.741092937, 
3164.030695499, 3691.009562725, 656.99925762, 2957.020256788, 
3504.589259205, 4571.053233048, 166.773860709, 4315.364099159, 
3562.893265958, 3220.426987198, 5425.646997566, 4478.730458262, 
2269.039013389, 5736.9136458, 5147.973979647, 4563.214559371, 
2435.627690299, 4949.432417698, 4874.889736798, 2447.370509174, 
3145.076913027, 1389.694649923, 3462.012459963, 4216.064489881, 
2950.829316152, 3407.14359775, 3208.884502676, 971.630172045, 
7833.251446154, 9276.179681786, 10191.588469976, 11739.8725632, 
873.051443516, 1001.923176224, 3454.355846919, 3425.274178718, 
3940.960628103, 3041.257518475, 3521.269460166, 3750.900953434, 
4439.542579025, 12680.947084854, 9733.258527423, 308.751777629, 
15041.183863368, 184.8452911, 9235.422525239, 11603.313679836, 
9513.435374799, 15661.892928366, 5.935499689, 6.483064133, 7.088646266, 
3.128106885, 3.453503382, 3.812565892, 1.743686801, 1.989766839, 
2.162768883, 1.18796692, 1.34776651, 1.460677952, 0.934184326, 
1.110563835, 1.22109392, 0.942966266, 1.059290854, 1.160741022, 
2249.187326262, 2406.920501236, 2372.969988995, 13885.052641784, 
3899.846831725, 21571.318729866, 20442.69884649, 15596.795756857, 
23269.331212058, 28003.045832026, 14391.00981935, 10991.588138466, 
12146.498955404, 18591.153622519, 19715.864962382, 18984.011787921, 
13495.39325162, 22933.268374506, 8932.297666899, 7905.304217295, 
13835.410007437, 15166.433852151, 25039.396226243, 19498.195086677, 
12877.101291995, 26668.263768903, 4233.761748941, 2327.605818745, 
4209.885290145, 3263.021192413, 1848.678983786, 2740.764688357, 
3522.095678614, 2643.799496697, 1027.961905777, 809.863513611, 
981.40024344, 449.54786886, 468.594754441, 611.794937399, 870.832204039, 
2588.091632841, 24184.621409418, 43768.721896141, 38754.006715268, 
13934.565982487, 4778.265090891, 35282.880906662, 25002.217878491, 
22070.171773063, 18451.139765434, 28212.058982346, 28104.822986025, 
27885.255536277, 14436.17849318, 15457.703118251, 11159.483503485, 
24741.803307462, 24782.255355859, 4396.614409775, 4112.328457279, 
1180.872397438, 403.233318452, 311.47541438, 2856.277573194, 
8.58561842, 2185.718471559, 9579.035204336, 4975.550069648, 2568.578937896, 
2089.908295499, 2637.423773557, 2284.730773573, 2393.81462317, 
1968.770922924, 2368.023842406, 2004.187214709, 2387.817824755, 
1937.455075614, 2721.312740164, 2364.053940896, 1144.36029701, 
981.097717693, 1121.308899158, 957.768545721, 1194.467741917, 
979.703759746, 1058.92901845, 955.306783423, 1122.185597996, 
934.796479528, 1087.446368377, 947.182890055, 1720.280668699, 
1396.44131266, 1264.822501925, 1066.467340058, 1129.872654755, 
978.591349084, 1024.847337793, 893.95641702, 1021.667396307, 
897.660025595, 1157.753772199, 987.559410695, 2001.341199852, 
1553.300242875, 1714.026538454, 1369.596672549, 1840.028672943, 
1514.597606925, 2261.869533177, 1742.980544272, 1217.472820174, 
1052.419994718, 2412.091237232, 1840.049556762, 2412.355425093, 
1833.902224002, 1166.150434442, 989.36654541, 1727.399223697, 
1379.823422282, 1521.437037284, 1307.698311662, 1281.978151486, 
1073.978176322, 839.110335868, 745.255285071, 971.298071042, 
822.368065555, 229.866035998, 192.151802573, 269.191494746, 226.679807394, 
1884.205897945, 1570.878395831, 1419.07307384, 1183.35502134, 
1344.128634438, 1115.750833426, 1245.67831164, 1141.48282157, 
1163.157347453, 995.399323402, 1158.509308703, 992.306084298, 
10555.60271354, 44028.286710071, 7319.269197947, 2295.729281756, 
3063.256792958, 3622.027748796, 749.772815644, 3249.538889951, 
3968.357168338, 4565.736226542, 160.991914129, 4385.588327515, 
3459.367581677, 3191.871637791, 5303.571737911, 4439.625936658, 
2397.84635726, 5455.774784214, 5127.936752666, 4607.134537924, 
2541.025856693, 5286.049572237, 4894.706322916, 2508.70503033, 
3181.774190885, 1374.073499192, 3483.986868181, 3980.942928815, 
2773.932650003, 3452.651932961, 2955.494692275, 992.741929243, 
8124.252045849, 10629.581729564, 9522.957665608, 11343.109807218, 
921.364421637, 1041.114150216, 3527.436450492, 3427.798606229, 
3922.490009706, 3102.330209642, 3573.750886853, 3924.736407354, 
4467.245524367, 12816.437197695, 9597.125378863, 304.582440239, 
14914.96392496, 267.127750039, 11706.146702234, 12514.145950955
)), class = c("spec_tbl_df", "tbl_df", "tbl", "data.frame"), row.names = c(NA, 
-1064L))
$\endgroup$
  • $\begingroup$ In $H_A$ do you mean "not all means equal"? Because $\ne$ isn't transitive, your current expression is meaningless. $\endgroup$ – BruceET May 22 at 19:40
  • 1
    $\begingroup$ With distributions this skewed and this close to each other, you will need very large amounts of data. How much to you have? $\endgroup$ – whuber May 24 at 18:50
  • 1
    $\begingroup$ See stats.stackexchange.com/questions/69898 for a discussion of data somewhat like yours in a similar setting (but with two groups instead of five). $\endgroup$ – whuber May 24 at 19:09
  • 1
    $\begingroup$ You can apply ANOVA in any event. I suspect it will be very conservative with such data in the sense of having a very hard time detecting a difference in mean values (which is what you are looking for when you're focusing on total computational time). The KW test doesn't really tell you much about differences in total computational time, but it might more readily detect that there is a difference of some sort among the groups. $\endgroup$ – whuber May 24 at 20:31
  • 1
    $\begingroup$ BTW, a similar situation with a more positive outcome (ANOVA worked well) is discussed at stats.stackexchange.com/questions/110418. $\endgroup$ – whuber May 24 at 20:34
1
+100
$\begingroup$

First things first

Maybe a simple solution is enough? Note that provided you have a reasonable number of measurements, you should be quite OK with just measuring the average total time and choosing the intervention that has the fastest one. The only thing a statistical analysis gives you on top of that is a) a new possibility to mess up and b) some assessment of uncertainty, i.e. whether you need to collect more data to be more certain. Given enough data even the tiniest difference will be significant. You are (I presume) making an engineering choice and not publishing a scientific paper.

For the following I assume you want to do statistics.

What do you care about?

It is also IMHO good idea to think about what difference are you actually interested in. Do you really care about the difference in mean performance? Or do you care about some sort of worst-case scenarios?

I would also suggest that for your use case, you might be better served by estimating the pairwise differences with some measure of uncertainty rather than by testing hypothesis - if you see that there is "no significant difference" betweeen treatments A and B it is way less informative than e.g. "95% CI for mean performance of A is (10;30), for B is (15;50) and for the difference A-B is (-45,18)" - in the latter case you are not certain which is better, but you see that B has higher risk of very large absolute times.

Bootstrap - dirty but cheap

A very simple but useful way to estimate uncertainty is the Bootstrap. A simple scheme you could use is to choose $N$ of the runs at random (with replacement) and compute all the quantities of interest (in your case probably all means and pairwise differences in means). Repeat this a lot - say 1000 times. This will get you 1000 "samples" of each quantity. You can than treat the middle 95% of the samples as your 95% uncertainty interval. $N$ could be equal to your actual number of runs or a bit smaller (say 0.95 * number of runs), you may want to test multiple values of $N$ and see if this affects results.

Modelling

The full-fledged (and most difficult) approach is to try to find a good model of your dataset.

My suggestion would be to try brms or other flexible linear model package. I think a student-t response with log link or a gamma response could have reasonably fat tails to not be led astray by the outliers.

Alternatively, you can try to find a transformation that makes your data look more normal and then fit a model with "normal" or "student" families (with identity link) - log(sqrt(x)) might be your friend, see https://en.wikipedia.org/wiki/Variance-stabilizing_transformation for some other ideas.

Using brms you can then use posterior predictive checks (PPC) to test whether your response distribution at least roughly matches your data (e.g. ppc_dens_overlay) and get posterior credible intervals for all the central tendencies and differences of interest, including (with some fiddling) differences in say 95% quantiles of runtimes, should you be interested.

I can elaborate a bit more if you think this is a sensible approach for you.

$\endgroup$
  • $\begingroup$ Thank you for your response. I have minimal experience in brms and Bayesian stats in general but very interested in pursuing this. I went ahead and ran fit1 <- brm(alive_time ~ intv, data = dat, family=student(link='log'). Since my independent vars are factors, I'm curious why I wouldn't use family=cumulative as that is suggested for ordinal models? I also ran pp_check(fit1) and my y_rep produced a bimodal distribution with no tails, while my y produced a right skewed normal with extremely fat tails. Also, should I be interested in Pop Level Effects of Family Specific Params? $\endgroup$ – dylanjm May 28 at 17:05
  • $\begingroup$ Also, my motivation is to show our new solvers are quicker than our previous solvers and will be reported in a milestone but not an academic paper. The first idea was to just compare means (which I suppose is still possible). I opened my loud mouth to say we should have some statistical measure to be sure they are truly better. I may have bitten off more than I can chew haha. $\endgroup$ – dylanjm May 28 at 17:10
  • 1
    $\begingroup$ @dylanjm The problem with statistics is that it is easy to get a number but often quite hard to make sure the number represents anything in the real world :-) A simple trick would be to use Bootstrap (added to the answer). Getting grip of linear models, while interesting and useful might not be the best investment of your time. But anyway: family=cumulative is when your response (here time taken) is ordered, but distances between neighbouring values may be variable (as in "Strongly agree" - ... - "Strongly disagree" questions). 1/N $\endgroup$ – Martin Modrák May 29 at 7:11
  • 1
    $\begingroup$ 2/2 if your pp_check shows strong disagreement betwen actual data (y) and what the model produced (yrep), something is wrong :-) I would check whether intv is treated as a factor (Pop Level Effects - which is mostly what you care about - have one entry for each but one of the levels of intv) and try other response families (gamma might also be reasonable). Or transform the data somehow (added to my answer). Note that there is no one-size-fits all solution, you need to understand your data and this may take non-trivial effort to debug. $\endgroup$ – Martin Modrák May 29 at 7:16
  • $\begingroup$ Seeing above to Ben's plots, do you think it would be reasonable to use a mixture prior of 3 approximately normal distributions on the log of my data and instead of doing differences as my beta parameter, remove the intercept and compute differences afterwards? Would this be overkill? $\endgroup$ – dylanjm May 30 at 16:56
3
$\begingroup$

I note from your comment that you have clarified that all the values of alive_time are positive time values, and the density plots in your post go into negative values only because they are kernel densities. In view of this, the first thing I would recommend is to plot your data on a logarithmic scale. This will make it easier to see the shape of your data over the five categories. In particular, it will probably make it easier to see what appears to be a secondary "hump" in the distribution.

Testing equality of means across groups is usually handled with ANOVA, but this generally relies on use of the CLT approximation to the distribution of the sample mean, and this can be sensitive to extremely high positive skew. (In the case where you have an underlying distribution with skewness so extreme as to give infinite variance, the CLT does not even apply.) The Kruskal-Wallis test is a one-way ANOVA on ranks of the data, so it involves a loss of information of the actual times. For this reason, it is probably a poor test if you are interested in equality of means.


Plotting the densities more clearly: By plotting the data on a logarithmic scale it will be easier to see how skewed this data is (e.g., relative to the lognormal distribution), and this will give you a reasonable idea of what you are dealing with. Since you have provided your data, here is some R code to plot the density functions on a logarithmic scale. I have made a plot of the stacked density plots, and another plot of the overlaid density plots. (In my code, the data frame you use is referred to as DATA, so you will need to create this object. I have also extracted the times for the individual interventions, and named these vectors with upper-case intervention names.)

#Load required libraries and set theme
library(ggplot2);
THEME <- theme(plot.title    = element_text(hjust = 0.5, size = 14, face = 'bold'),
           plot.subtitle = element_text(hjust = 0.5, face = 'bold'));

#Plot stacked densities on logarithmic scale
ggplot(data = DATA, aes(x = alive_time)) + 
    geom_density(fill = 'blue') + facet_wrap(~ intv, ncol = 1) + THEME +
    expand_limits(x = c(10^(-1), 10^6)) + 
    scale_x_log10(breaks = scales::trans_breaks("log10", function(x) 10^x),
         labels = scales::trans_format("log10", scales::math_format(10^.x))) +
    ggtitle('Kernel Density Plots') + xlab('Alive Time') + ylab('Density');

#Plot overlayed densities on logarithmic scale
ggplot(data = DATA, aes(x = alive_time, colour = intv)) + 
    geom_density(size = 1) + THEME +
    labs(colour = 'Intervention') +
    expand_limits(x = c(10^(-1), 10^6)) + 
    scale_x_log10(breaks = scales::trans_breaks("log10", function(x) 10^x),
         labels = scales::trans_format("log10", scales::math_format(10^.x))) +
    ggtitle('Kernel Density Plots') + xlab('Alive Time') + ylab('Density');

enter image description here

enter image description here

You can see from these plots that the five intervention groups have very similar densities for the log-time. Visually, this suggests that there are only minor differences in the distributions, if any. We can test this formally using appropriate statistical tests.


Testing for a common distribution: In this case we can formally test for a common distribution using a multivariate Anderson-Darling test. This is a non-parametric test that does not assume any particular distributional form, so it is appropriate for this case, where the density does not conform to any common distributional family.

#Load required libraries and set theme
library(ggplot2);
library(kSamples);

#Apply Anderson-Darling Test
#The vectors DANIELL, ... , BASE were defined by extracting alive times for interventions
TEST <- ad.test(DANIELL, SHANEL, TIMESTEP, AD, BASE);
TEST;

 Anderson-Darling k-sample test.

Number of samples:  5
Sample sizes:  210, 211, 211, 213, 219
Number of ties: 0

Mean of  Anderson-Darling  Criterion: 4
Standard deviation of  Anderson-Darling  Criterion: 1.51789

T.AD = ( Anderson-Darling  Criterion - mean)/sigma

Null Hypothesis: All samples come from a common population.

              AD    T.AD  asympt. P-value
version 1: 3.196 -0.5296           0.6597
version 2: 3.200 -0.5294           0.6595

From this output we see that the test shows no evidence of a difference in the distributions (i.e., we do not reject the null hypothesis of a common distribution). That is, it is plausible that all the data from the five interventions comes from a common underlying distribution.

$\endgroup$
  • $\begingroup$ Hi Ben, thanks for the response. I went ahead and edited my question to include my data and a plot of the alive_time on a log10 scale. You were right about a 2nd hump in the data and now it appears to be more skewed left at the smaller values closer to one. I appreciate any insight as to what I could do to quantify differences in these run-times. Keep in mind that I'm currently using proxy-data, which will closely resemble this data's distribution, while I wait for the simulations to finish. $\endgroup$ – dylanjm May 29 at 14:57
  • $\begingroup$ I have updated the post to do some analysis on the data you provided. $\endgroup$ – Reinstate Monica May 29 at 23:17
  • $\begingroup$ Nice work with the plots. Does the Anderson-Darling test give me some some estimate on what is the smallest between-group difference that was ruled out? Or can any kind of "power" (in whatever sense) defined? Not rejecting equality is IMHO not very interesting on its own from a practical standpoint as it may just mean we have too few data points. Obviously we have the plots as some sort of estimate, so this might not be a big deal... Just curious. $\endgroup$ – Martin Modrák May 30 at 6:17
  • $\begingroup$ Ben, thanks for the in-depth response. I really like this idea. I'm curious, is there something similar to what would be a TukeyHSD test to determine which distributions were most different? Anyway I can quantify further the differences between the cases? I looked into something called KL Divergence which seems close to this idea. I greatly appreciate the time and effort both you and @Martin have spent on this. $\endgroup$ – dylanjm May 30 at 16:52
  • $\begingroup$ @dylanjm: There are a whole bunch of nonparametric tests that test for equality of distributions. They all involve using some measure of difference between distributions, and then comparing this outcome to its null distribution under the null hypothesis of identical distributions. Some use KL divergence, and some use other measures of difference. If you are interested in this topic, it would be worth reading a book about nonparametric statistics, because it is a pretty big field. $\endgroup$ – Reinstate Monica May 30 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.