I am reviewing the statistical analysis of "Debugging Tests for Model Explanations" in arXive.
In the paper, the authors have subjects looking at the output of 5 different ml models.
In each of these 5 models they use 3 explanation techniques. On each model with each explanation technique, the same participants make a rating on a 5-point scale.
For analysis, the authors split the data by the 5 different models and analysed the ratings for each using a one-way ANOVA with the explanation technique as the only factor.
Besides, it is questionable whether to use ANOVA on self-reported rating scales; this is a two-factorial design for me. I would have used two-way repeated measures ANOVA.
My questions are:
- Is the way the authors analysed their data valid if they wanted to answer questions like: For model 1 explanation technique A receive significantly higher ratings?
- Wouldn't they have needed to correct for multiple comparisons when running multiple ANOVAs on data obtained from the same subjects?
- Is there any harm done by ignoring the second factor and the fact that this is a within-subject design?