I am reviewing the statistical analysis of "Debugging Tests for Model Explanations" in arXive.

In the paper, the authors have subjects looking at the output of 5 different ml models.

In each of these 5 models they use 3 explanation techniques. On each model with each explanation technique, the same participants make a rating on a 5-point scale.

For analysis, the authors split the data by the 5 different models and analysed the ratings for each using a one-way ANOVA with the explanation technique as the only factor.

Besides, it is questionable whether to use ANOVA on self-reported rating scales; this is a two-factorial design for me. I would have used two-way repeated measures ANOVA.

My questions are:

  1. Is the way the authors analysed their data valid if they wanted to answer questions like: For model 1 explanation technique A receive significantly higher ratings?
  2. Wouldn't they have needed to correct for multiple comparisons when running multiple ANOVAs on data obtained from the same subjects?
  3. Is there any harm done by ignoring the second factor and the fact that this is a within-subject design?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.