One vs all statistical test

Question

I have 3 groups of samples e.g. Group A, Group B, and Group C. I want to find out whether the mean of Group B and C are equal and different from Group A. Which statistical test should I use? All groups are of different small sizes (from 3 to 15). Is this valid to perform Welch's t-test considering Group B and C as a single group (though they are not the same. Both sample size and variance are different).

Answer 1

If data in each of your three groups are from normal populations, then you can begin with an ANOVA that does not assume equal variances, to see if there are any significant differences at all among sample group means $\bar A, \bar B$ and $\bar C.$

If so, you can do ad hoc tests to see if there is a significant difference between B and C. If B and C do not differ significantly, then you can compare A with B and A with C.

Because you say the groups have different variances, those comparisons could be done using Welch two-sample t tests. In order to avoid 'false discovery' doing multiple tests on the same data, each of these ad hoc tests could be at the 1.6% level according to the Bonferroni method.

I would be reluctant to combine B and C even if you find no significant difference between them. Failure to find a significant difference does not guarantee there is no difference. Moreover, you say groups have differing variances; if B and C have different variances, that's another reason not to treat them as the same.

Here are simulated data that may be sufficiently similar to yours to serve as an illustration.

 set.seed(2020)
 a = rnorm(10, 53, 2)
 b = rnorm(15, 60, 4)
 c = rnorm( 6, 62, 3)
 x = c(a,b,c);  g = rep(1:3, c(10,15,6))

A stripchart (dotplot) of the data shows the 31 individual observations in the three groups.

 stripchart(x~g, ylim=c(.5,3.5), pch=20)

 oneway.test(x ~ g)


        One-way analysis of means 
     (not assuming equal variances)

data:  x and g
F = 33.977, num df = 2.000, denom df = 16.606, 
  p-value = 1.351e-06

In R, the procedure oneway.test does does not assume that groups have equal variances. With P-value near $0,$ we have very strong evidence that there is at least one significant difference among the three group means. We do ad hoc Welch t tests (still not assuming equal group variances), to see which differences are significant. [Here we show only the P-values of the three Welch tests; remove $p.val from each line of code to see the full output.]

t.test(b,c)$p.val
[1] 0.2962728
t.test(a,b)$p.val
[1] 0.0001184109
t.test(a,c)$p.val
[1] 3.772418e-06

There is no significant difference between groups B an C. Looking at the stripchart above for my simulated data, I would be reluctant to combine B and C into a group. While their sample means are not significantly different, it is not clear that they were sampled from the same population. (Of course, looking at my R code to simulate the samples, you could verify the differences between the two populations; of course, in a real application there would be no way to know for sure whether they come from different populations.)

However, A is significantly different from B and also from C. Both of these ad hoc tests have very small P-values, so there seems no danger of false discovery.

Note: Others who have commented here have somehow assumed that your groups were not sampled from normal populations. With such small sample sizes there would be no way to be sure (one way or the other) about that. And I see no mention of nonnormality in your question.

If you have reason to believe that populations for your kind of work are not normal, you might use a Kruskal-Wallis test instead of the initial ANOVA, and use two-sample Wilcoxon (rank sum) tests for ad hoc comparisons.

Both of these tests (K-W and 2-sample Wilcoxon) assume that the three populations have the same shape, but that their positions may be shifted relative to one another. Because same shape should be taken to include equal variances, that would be a weakness in an analysis depending on these nonparametric tests.

Answer 2

If you assume that groups B and C have same means (which you can check easily), then you can group them to a group BC (with still the same mean) and use a non parametric test such as Mann–Whitney-Wilcoxon test (wilcox.test on R) to compare this group BC with group A. This test is testing equality of medians rather than equality of the means, but this is usually still relevant.