I'm assessing a software tool that tries to repair a specific type of error in the source code of programs. In particular, I'm comparing two different configurations of this tool. The tool is assessed in a within-subject experiment designed as follows:

  • 290 errors collected from 3 different target systems (each error is one subject);
  • 2 treatments (configuration "A" and configuration "B");

For each error, I run the tool in both configurations and compare the effectiveness score measured in each configuration. Then I use a Wilcoxon signed-rank test to test the null hypothesis that there is no significant difference between the treatments.

My question is: when I perform the hypothesis testing, should I consider all errors together as one single sample? Or should I split the data per target system from where the errors were collected?


In this case, it would seem reasonable to test for an interaction between target system and treatment on the effectiveness score. A basic way to do this is to use a two-way ANOVA (provided the necessary assumptions are satisfied); in other words, we consider the linear model $$\text{EFFECTIVENESS} = \text{SYSTEM} + \text{TREATMENT} + \text{SYSTEM}*\text{TREATMENT}$$ and test the null hypothesis that the coefficients corresponding to $\text{SYSTEM}*\text{TREATMENT}$ are all zero. If this null hypothesis is rejected, then that means we have detected a significant interaction, and it would make sense to perform a separate analysis for each target system. On the other hand, if no significant interaction is detected, then we would consider the simpler model $$\text{EFFECTIVENESS} = \text{SYSTEM} + \text{TREATMENT}$$ At this point, we could test whether the system is a significant factor (i.e., whether we reject the null hypothesis that its coefficients are all zero), and if not, we could discard it, in which case we can lump the data together from all the systems, and just apply a two-sample procedure to test whether there is a significant difference between treatments. But if the system remains a significant factor, then can test whether the treatment is a significant factor by comparing the previous model with the nested model $$\text{EFFECTIVENESS} = \text{SYSTEM}$$ using an F test.

If a normal distribution is not a good fit for the residuals, then the preceding approach can be adapted using a generalized linear model; for example, if the effectiveness score is based on a count of the number of errors that are successfully repaired, then Binomial regression may be appropriate.

| cite | improve this answer | |
  • $\begingroup$ What about the case where the sample does not meet the assumptions of parametric tests? Is there a non-parametric equivalent? $\endgroup$ – EijiAdachi Oct 14 '15 at 17:35
  • $\begingroup$ You may want to consider whether first applying a transformation to your data could bring it into a form where a parametric test applies. But if that fails, you could look into non-parametric regression techniques (this may be a useful starting point: socserv.socsci.mcmaster.ca/jfox/Courses/Oxford-2005/index.html). $\endgroup$ – Brent Kerby Oct 16 '15 at 23:31
  • $\begingroup$ I actually tried to apply log and square root transformations, but even so, the normality and homoscedasticity assumptions were not met. Thanks for the reference. I'll take a look. $\endgroup$ – EijiAdachi Oct 22 '15 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.