# In a within-subject experiment, should I split my data or not?

I'm assessing a software tool that tries to repair a specific type of error in the source code of programs. In particular, I'm comparing two different configurations of this tool. The tool is assessed in a within-subject experiment designed as follows:

• 290 errors collected from 3 different target systems (each error is one subject);
• 2 treatments (configuration "A" and configuration "B");

For each error, I run the tool in both configurations and compare the effectiveness score measured in each configuration. Then I use a Wilcoxon signed-rank test to test the null hypothesis that there is no significant difference between the treatments.

My question is: when I perform the hypothesis testing, should I consider all errors together as one single sample? Or should I split the data per target system from where the errors were collected?

In this case, it would seem reasonable to test for an interaction between target system and treatment on the effectiveness score. A basic way to do this is to use a two-way ANOVA (provided the necessary assumptions are satisfied); in other words, we consider the linear model $$\text{EFFECTIVENESS} = \text{SYSTEM} + \text{TREATMENT} + \text{SYSTEM}*\text{TREATMENT}$$ and test the null hypothesis that the coefficients corresponding to $\text{SYSTEM}*\text{TREATMENT}$ are all zero. If this null hypothesis is rejected, then that means we have detected a significant interaction, and it would make sense to perform a separate analysis for each target system. On the other hand, if no significant interaction is detected, then we would consider the simpler model $$\text{EFFECTIVENESS} = \text{SYSTEM} + \text{TREATMENT}$$ At this point, we could test whether the system is a significant factor (i.e., whether we reject the null hypothesis that its coefficients are all zero), and if not, we could discard it, in which case we can lump the data together from all the systems, and just apply a two-sample procedure to test whether there is a significant difference between treatments. But if the system remains a significant factor, then can test whether the treatment is a significant factor by comparing the previous model with the nested model $$\text{EFFECTIVENESS} = \text{SYSTEM}$$ using an F test.