# Evaluating software performance to identify regressions resulting from source code changes

I am attempting to monitor the performance of a software package in order to identify when a change to the code base introduces a performance regression (a slow down in the code). My assumption is that if the performance of the data is unchanged, I'm essentially collecting samples from the same population, and I should be able to use a wilcoxon signed-rank test to test my null hypothesis.

To test out this process, I collected 100 samples of performance data for a test application, each containing 1000 observations. For each sample, I compared it to each of the other samples using Python's scipy.stats.wilcoxon function, for example:

Sample    Sample    Wilcoxon test    p-value
0         1         228976           0.019
0         2         227054           0.011
0         3         237746           0.171
...


Then, for each sample, I counted the number of comparisons for that sample for which the p-value was less than my target threshold (0.05). I would have expected ~5% of the samples to be less than this threshold, however in my observations this was typically closer to ~30%, for example:

Sample    # comparison p-values < 0.05
0         28
1         36
2         40
3         25
...


This seems to imply that over 30% of the samples are showing a statistically significant performance difference, even though the performance should be essentially the same (the same code is executed in each instance). I'm attempting to understand what in my experiment could lead to that sort of result, and find out if there is a way to correct it. Or alternatively, if there is a more effective way to determine if a slow down as observed in a software performance test is significant.

• You could be mis-applying the test: your description is vague. Could you clarify it? The performance could be varying in non-random ways, too, and maybe you are detecting that. Moreover, by comparing 100 samples to each other, you could get more significant results than you anticipated--but the proportion won't rise from 5% all the way to 30%. – whuber Sep 16 '13 at 22:12
• @whuber Thanks for the feedback, I've updated the description to (hopefully) clarify the test that I'm running. I think you're correct in that I'm detecting differences that aren't related to randomness, but might be related to load on the machine or other factors. Are there ways to attempt to account for this in the data analysis? – DRH Sep 17 '13 at 0:05
• Are you controlling the load on the test environment? If the baseline load of your environment is changing from one test to another, you're not going to get great results here unless the changes in the code base are causing massive changes in performance. – David Marx Sep 17 '13 at 5:50
• @DavidMarx Typically yes, the only active processes on the test machine should be the performance test, however there may still be some background activity from the OS. – DRH Sep 17 '13 at 19:16

I am not privileged enough to comment, yet, therefore this as an answer:

If I get this correctly, then you compare pairs of vectors using the Wilcoxon-signed rank test. This is a test for paired observations. This would make sense if the observations compared are conceptually linked. now the question is: Is your problem represented by the following simulation in Matlab (apart from the generation of random data from a normal distribution as used here)?

clear p;
clear psig;
step_i=0;
a= normrnd(0,1,100,1000);

for t_i= 1:99
for t_j=t_i+1:100
step_i=step_i+1
p(step_i)=signrank(a(t_i,1:1000), a(t_j,1:1000));
[t,g(step_i)]=ttest2(a(t_i,1:1000), a(t_j,1:1000));
end
end

psig= p<0.05;
gsig=g<0.05;
pmean=mean(psig,1);
pmeantotal=mean(pmean);

gmean=mean(gsig,1);
gmeantotal=mean(gsig);


Here I find only the expected deviations from 5% significant results both for the Wilcoxon and the ttest in several runs. Therefore it has to be either the data generation, the data matching procedure or a coding problem that explains your results. It might be interesting to seee whether the non-randomness is evenly dispersed or whether there is a specific group of runs with a difference in central tendency, etc.