Accuracy vs Precision: Can I use the Sphercity Assumption data in a meaningful way?

Question

I understand that when conducting a repeated measures design with say one factor having three levels (Training: None, Standard, New Method), in the analysis you want the sphercity assumption to not be violated, i.e. that the variability of responses of one participant are consistent throughout the different repeated conditions. But, isn't just applying a correction if this assumption is not met ignoring potentially useful information?

This is my hypothetical: say that the training I am running has an effect. In Standard training everyone has 10points higher than in the None, but when they are given the New Method, some get extremely better, 50points, while some only 15points. If I just look at the means, i see that None gives a performance on average of 100, Standard training of 110, and New Method of 130. If Sphericity is violated, then I just correct the DF and report that. However, isn't the fact that the New Method, while showing better scores overall, is also wildely less precise (or unpredictable) in performance boost important?

TL;DR, if a manipulation has a significant effect, but the variability of this effect is much higher than that of previous manipulations, can I still somehow report that information in a meaningful way?

Answer 1

From your brief description, it seems you may have data somewhat like the ones I have simulated below (in R). My 'New' group looks as if it may not not normal, so use of a standard one-factor ANOVA may not be advisable.

set.seed(1212)  # for reproducibility
None = round(rnorm(20, 100, 15))
Std =  round(rnorm(20, 110, 10))
New =  round(rexp(20, 1/30) + 100)
x = c(None, Std, New);  g = rep(1:3, each=20)

Here are boxplots and stripcharts of these data:

It seems clear that 'New' scores are generally higher than scores for the other two groups. A Kruskal-Wallis nonparametric test detects differences among the groups.

kruskal.test(x ~ g)

       Kruskal-Wallis rank sum test

data:  x by g
Kruskal-Wallis chi-squared = 17.321, df = 2, p-value = 0.0001733

Now we use two-sample Wilcoxon tests to check for pair-wise differences among the groups. Using the Bonferroni method of protecting against 'false discovery' we will not declare significant differences unless P-values are smaller than about 1.5%.

According to this criterion, we do not find 'None' and 'Std' to be significantly different (P-value about 26%), but we do find 'Std' and 'New' to be different (P-value about 0,05%). [Not surprisingly, a third Wilcoxon test, not shown, finds 'None' and 'New' to be different (P-value about 0.04%).]

wilcox.test(None, Std)

        Wilcoxon rank sum test with continuity correction

data:  None and Std
W = 157.5, p-value = 0.2553
alternative hypothesis: true location shift is not equal to 0

Warning message:
In wilcox.test.default(None, Std) : cannot compute exact p-value with ties


wilcox.test(Std, New)

        Wilcoxon rank sum test with continuity correction

data:  Std and New
W = 70.5, p-value = 0.0004772
alternative hypothesis: true location shift is not equal to 0

Warning message:
In wilcox.test.default(Std, New) : cannot compute exact p-value with ties

Notes: (1) I deliberately rounded my fake data to integers, in order to get some ties. Unless there are very many ties, the Wilcoxon test in R gives reasonably accurate P-values. If there are very many ties (especially between groups) a permutation test can be used instead.

(2) I will leave it to you to read about Kruskal-Wallis, and Mann-Whitney-Wilcoxon tests. Also, to read about the Bonferroni (and other methods) of avoiding false-discovery in 'post hoc' testing. Alternative terminology is to 'control the family error rate'.

(3) Of course I'm speculating here based on your brief description of your data. I don't claim that my fake data really emulate yours, so you may find very different results comparing the three groups. However, the methods I have shown should be useful. If you still have doubts about analysis of your data, you might post them in a fresh Question, referring to this Answer and your specific difficulties.