I am attempting to monitor the performance of a software package in order to identify when a change to the code base introduces a performance regression (a slow down in the code). My assumption is that if the performance of the data is unchanged, I'm essentially collecting samples from the same population, and I should be able to use a wilcoxon signed-rank test to test my null hypothesis.
To test out this process, I collected 100 samples of performance data for a test application, each containing 1000 observations. For each sample, I compared it to each of the other samples using Python's
scipy.stats.wilcoxon function, for example:
Sample Sample Wilcoxon test p-value 0 1 228976 0.019 0 2 227054 0.011 0 3 237746 0.171 ...
Then, for each sample, I counted the number of comparisons for that sample for which the p-value was less than my target threshold (0.05). I would have expected ~5% of the samples to be less than this threshold, however in my observations this was typically closer to ~30%, for example:
Sample # comparison p-values < 0.05 0 28 1 36 2 40 3 25 ...
This seems to imply that over 30% of the samples are showing a statistically significant performance difference, even though the performance should be essentially the same (the same code is executed in each instance). I'm attempting to understand what in my experiment could lead to that sort of result, and find out if there is a way to correct it. Or alternatively, if there is a more effective way to determine if a slow down as observed in a software performance test is significant.