A one-way ANOVA for multiple comparison

Question

Cheng, H., & Liu, S. (2019). Haptic Force Guided Sound Synthesis in Multisensory Virtual Reality (VR) Simulation for Rigid-Fluid Interaction. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 111–119). IEEE. https://doi.org/10.1109/VR.2019.8797906

In this paper, Section 5.3 presents several experiments to assess the effectiveness of their approach. They asks three questions for each scenarios with or without audio feedback:

1. Can you recognize that you are interacting with the water, despite the rigid lever provides conflicting perceptual cues?

2. Do you think increasing haptic feedback / haptic and audio feedback can increase reality?

3. How strong is the sense of reality?

For the first and second question, the participant answer yes or no, so that they record their response as 1 or 0. The last question is the satisfaction score from 1 to 10.

I thought that they should compare the results by using a paired t-test to each question result, because the condition is related to 'audio'.

In their result, they calculate the mean and variance for each question, and compute p-value and t-value for each scenario. (I think that they find the p-value and t-value by comparing two conditions)

(One scenario with audio, Mean (Q1): 0.73, Std (Q1): 0.2095, Mean (Q2): 0.67, Std (Q2): 0.2381, Mean (Q3): 6.33, Std (Q3): 2.5238),

(One scenario without audio, Mean (Q1): 0.33, Std (Q1): 0.2381, Mean (Q2): 0.40, Std (Q2): 0.2571, Mean (Q3): 3.47, Std (Q3): 1.4095)

P-value: 6.431, T-value: < 0.005

They find the statistical significance between two condition by using a one-way Analysis of Variance (ANOVA) test, and they get the reported p-value is less than a significance level of 0.05.

Moreover, they apply the Post-hoc analysis (pairwise comparison between conditions) by using Bonferroni method and Wilcoxon signed-rank test.

How they the p-value and t-value based on this experiment setting? Is their method valid?

Answer 1

There seems to be a number of problems with the presentation of results and the statistics in Section 5.3 of the referenced paper:

1) The results are presented in rather poor English so it is hard to judge what exactly the authors mean. (Admittedly, mine isn't perfect either, but this section is close to incomprehensible.)

2) As sysexits mentions: It is strange that multiple one-way-ANOVAs are performed to test for pairwise differences between two levels of the factor audio/no audio, since an one-way ANOVA is commonly used to test for differences in more than two factor levels. However, this should not be a big problem: In principle, a one-way ANOVA with only two factor levels should yield similar results as a t-test, if variances are comparable across the two conditions (see this discussion as a ref).

3) ANOVA results are provided with t-Values, which is also strange since the ANOVA is a F-test, so we would expect F-values. I guess this is due to the fact that the authors present the results of mentioned "post-hoc" pairwise comparisons, which is often provided by default by stats-software when performing an one-way ANOVAs. Since there is only two conditions, this should not be a problem either, as the ANOVAS and the t-test should yield roughly the same results.

Now to the more worrying points:

4) Table 3 presents only one test result for three different outcome measures (Q1, Q2, Q3), over two conditions, for multiple scenarios. This makes me wonder what was tested here. Either a) a different test was performed than described, b) testing results are presented for only one, not three outcomes or c) (most likely) the authors have no clue what they are doing.

5) It seems that all testing was performed within the same 15 volunteers (but description is too poor to judge with certainty). It is not mentioned whether a standard independent-sample one-way ANOVA or a repeated-measures ANOVA was performed. Since there is no explicit statement about this, I guess the authors did the standard ANOVA. If repeated measures were not accounted for, the assumption of independence was violated and test results invalid.

6) The biggest problem is that Q1 and Q2 are dichotomous variables (yes/no), which do not follow a normal distribution by definition. Therefore a main assumption of both t-Tests and one-way ANOVA was violated, tests are therefore invalid. Arguably this is also the case for Q3 (10 level ordinal variable). Non-parametric tests would be more appropriate.

All in all, it seems that inferential testing was presented and performed poorly in the paper mentioned. However, judging from the descriptive results, the differences between audio/no audio conditions in Experiment 2 are large enough (compared to the SDs) that appropriate statistical tests would probably yield statistically significant results, too.