Evaluating software performance to identify regressions resulting from source code changes I am attempting to monitor the performance of a software package in order to identify when a change to the code base introduces a performance regression (a slow down in the code). My assumption is that if the performance of the data is unchanged, I'm essentially collecting samples from the same population, and I should be able to use a wilcoxon signed-rank test to test my null hypothesis.
To test out this process, I collected 100 samples of performance data for a test application, each containing 1000 observations. For each sample, I compared it to each of the other samples using Python's scipy.stats.wilcoxon function, for example:
Sample    Sample    Wilcoxon test    p-value   
0         1         228976           0.019
0         2         227054           0.011
0         3         237746           0.171
...

Then, for each sample, I counted the number of comparisons for that sample for which the p-value was less than my target threshold (0.05). I would have expected ~5% of the samples to be less than this threshold, however in my observations this was typically closer to ~30%, for example:
Sample    # comparison p-values < 0.05
0         28
1         36
2         40
3         25
...

This seems to imply that over 30% of the samples are showing a statistically significant performance difference, even though the performance should be essentially the same (the same code is executed in each instance). I'm attempting to understand what in my experiment could lead to that sort of result, and find out if there is a way to correct it. Or alternatively, if there is a more effective way to determine if a slow down as observed in a software performance test is significant.
 A: I am not privileged enough to comment, yet, therefore this as an answer:
If I get this correctly, then you compare pairs of vectors using the Wilcoxon-signed rank test. This is a test for paired observations. This would make sense if the observations compared are conceptually linked.
now the question is: Is your problem represented by the following simulation in Matlab (apart from the generation of random data from a normal distribution as used here)? 
clear p;
clear psig;
step_i=0;
a= normrnd(0,1,100,1000);

for t_i= 1:99
    for t_j=t_i+1:100
        step_i=step_i+1
p(step_i)=signrank(a(t_i,1:1000), a(t_j,1:1000));
[t,g(step_i)]=ttest2(a(t_i,1:1000), a(t_j,1:1000));
    end
end

psig= p<0.05;
gsig=g<0.05;
pmean=mean(psig,1);
pmeantotal=mean(pmean);

gmean=mean(gsig,1);
gmeantotal=mean(gsig);

Here I find only the expected deviations from 5% significant results both for the Wilcoxon and the ttest in several runs. Therefore it has to be either the data generation, the data matching procedure or a coding problem that explains your results. It might be interesting to seee whether the non-randomness is evenly dispersed or whether there is a specific group of runs with a difference in central tendency, etc. 
