Which test to use for comparing changes in computer source code?

Question

I am not sure which test I have to use to analyze my data. The situation is as follows:

I have two computer programs which are doing exactly the same. Both share a large amount of source code, but some parts were developed with different technologies. I want to test the effect of the different technology. I performed the same changes on both programs and measured some source code properties after every change. The changes were performed in a chain. I performed change 2 on the result of performing change 1 on the initial version, and so on.

Now, with the help of the properties measured after every change, I want to test if the program which was written with technology A were easier to change than the program written with technology B.

Which statistical test do you think is the best choice? I thought of the Student's T-Test for paired samples or the Wilcoxon signed-rank test. But I am unsure if theses tests are applicable to this situation.

Answer 1

Superficially, the Student's t-test for paired samples would be a good choice, if the assumption of normality is met for the distribution of your source code measurements. If it isn't, then the Wilcoxon signed-rank test would be a good next step. However, there is a more subtle assumption of both tests that is likely violated here - that the observations are independent from one another. In other words, the source code properties after change 1 are independent of the source code properties after change 2. I've never measured source code properties before, but this seems unlikely to me.

One way to improve the situation would be to look at the change in source code properties from one step to the next. This would get rid of any "baseline" differences causing correlations, but still doesn't help with e.g. an improvement in step 2 making the changes after steps 3 and 4 bigger.

Dealing directly with the autocorrelation in the observations takes you into the realm of mixed models or perhaps time series analysis - it is not a question of a simple test statistic anymore.

Answer 2

It depends. The right test to use will depend upon the nature of the value you are measuring. What is the output of your measurement? Is it a real number? An integer? A boolean? Also, the choice of statistical test will depend upon the distribution of these measurements. Is there a reason to expect these measurements to be approximately normally distributed? When you chart the distribution of your measurements, do they appear to be normally distributed?

Generally speaking: I would use the Wilcoxon signed-rank as the default choice for your situation. It does not assume normality. If you know that your measurements are normal, then you could use the Student's t-test for paired samples.

However: That said, it sounds like atiretoo has correctly diagnosed some ways in which these tests are inappropriate for your situation. Therefore, the right answer is probably that you should not use any of these tests, and you should go back to the drawing board and re-think the experimental design.