# Which test to use for comparing changes in computer source code?

I am not sure which test I have to use to analyze my data. The situation is as follows:

I have two computer programs which are doing exactly the same. Both share a large amount of source code, but some parts were developed with different technologies. I want to test the effect of the different technology. I performed the same changes on both programs and measured some source code properties after every change. The changes were performed in a chain. I performed change 2 on the result of performing change 1 on the initial version, and so on.

Now, with the help of the properties measured after every change, I want to test if the program which was written with technology A were easier to change than the program written with technology B.

Which statistical test do you think is the best choice? I thought of the Student's T-Test for paired samples or the Wilcoxon signed-rank test. But I am unsure if theses tests are applicable to this situation.

-

Superficially, the Student's t-test for paired samples would be a good choice, if the assumption of normality is met for the distribution of your source code measurements. If it isn't, then the Wilcoxon signed-rank test would be a good next step. However, there is a more subtle assumption of both tests that is likely violated here - that the observations are independent from one another. In other words, the source code properties after change 1 are independent of the source code properties after change 2. I've never measured source code properties before, but this seems unlikely to me.

One way to improve the situation would be to look at the change in source code properties from one step to the next. This would get rid of any "baseline" differences causing correlations, but still doesn't help with e.g. an improvement in step 2 making the changes after steps 3 and 4 bigger.

Dealing directly with the autocorrelation in the observations takes you into the realm of mixed models or perhaps time series analysis - it is not a question of a simple test statistic anymore.

-
You are right, the observations are not independent from one another. Is there a test that doesn't require independence? –  Apfelsaft Jul 7 '12 at 14:58
@Apfelsaft, no, I don't think so. I think atiretoo gave good advice for how to reduce the correlation, but the right answer is to change your experimental design. If the experimental design isn't right, it is very hard to compensate for that through choice of statistical test. In this case, a better experimental design might be to use a different original program for each trial, and apply one change to that program, to reduce correlation. –  D.W. Jul 7 '12 at 23:43