To begin with, I must admit, I am completely new to statistics. There are many discussions on CrossValidated on the optimal ways to compare the central tendency of two independent populations, which violate normality and homogeneity assumptions. Among the most useful threads I may cite the following: Mann Whitney test with unequal variances; Unequal variances t-test or U Mann-Whitney test?; Reporting Mann-Whitney U-Test without homogenity of variance; How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples; Equivalence of Mann-Whitney U-test and t-test on ranks; and Bootstrap vs. permutation hypotheis testing . Following the arguments and references put forward in these treads (and cited in the introductive works, available to me), I fell under the impression that there is no single approach for solving the problem, but rather different tests demonstrate different levels of power under different conditions (as is argued by Fagerland and Sandvik 2009 (doi 10.1016/j.cct.2009.06.007)) and suit different purposes (Greg Snow in Bootstrap vs. permutation hypotheis testing).
I am an archaeologist. I am testing the hypothesis whether objects of a certain type displaying a certain property (boolean factor JPXB in the examples below) tend to be higher than objects of the same type without this property. Thus, I have two groups in my population (which is a possibly exhaustive catalogue of objects of this type known today). The heights in both groups violate normality (transformations do not help) and their distributions are different.
Group 1: n=54, median=54.00cm, mean=61.19cm, range=108cm, std.dev=23.76cm, Shapiro-Wilk W=0.88 with p=5.69⋅10−5, skewness=1.22;
Group 2: n=789, median=40.00cm, mean=41.06cm, range=112cm, std.dev=14.90cm, Shapiro-Wilk W=0.94 with p=1.2⋅10−16, skewness=1.09.
That both groups deviate from normality is evident from QQ-plots, and that the groups are different is confirmed by Levene test F=10.66, p=0.0011.
The different tests proposed in the literature and in discussions on CrossValidated for such data return the same results.
1) The one-sided Welch t-test, which is claimed by some to be reasonably robust to perform well on large enough samples violating normality (see, particularly the CrossValidated threads What normality assumptions are required for an unpaired t-test? And when are they met?, How robust is the independent samples t-test when the distributions of the samples are non-normal? and How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples ), which should be my case with n=54 and 789, returns p=4.49⋅10−8.
>t.test(Height~factor(JPXB), data=LMK, alternative="greater")
Welch Two Sample t-test
data: Height by factor(JPXB)
t = 6.1419, df = 55.888, p-value = 4.492e-08
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
14.64681 Inf
sample estimates:
mean in group -1 mean in group 0
61.18519 41.05703
2) The one-sided Welch t-test of ranks (referred to as the Welch U test in Fagerland and Sandvik 2009 (doi: 10.1016/j.cct.2009.06.007) and considered inappropriate in the replies to the thread Equivalence of Mann-Whitney U-test and t-test on ranks), returns p=2.05⋅10−13.
> t.test(rank(Height)~factor(JPXB), data=LMK, alternative="greater")
Welch Two Sample t-test
data: rank(Height) by factor(JPXB)
t = 9.0288, df = 65.557, p-value = 2.048e-13
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
197.0243 Inf
sample estimates:
mean in group -1 mean in group 0
648.2037 406.5184
3) The Yuen t-test with 20% trimming returns p=2.72⋅10−6.
> library(PairedData)
> yuen.t.test(Height~factor(JPXB), data=LMK, tr = 0.2, alternative="greater")
Two-sample Yuen test, trim=0.2
data: Height by factor(JPXB)
t = 5.3531, df = 35.169, p-value = 2.716e-06
alternative hypothesis: true difference in trimmed means is greater than 0
95 percent confidence interval:
11.30603 Inf
sample estimates:
trimmed mean in group -1 trimmed mean in group 0
56.38235 39.86316
4) The Mann–Whitney test (which should in the case of non-homogeneous samples provide the evidence of stochastic dominance, and not measure the location shift, so Reporting Mann-Whitney U-Test without homogenity of variance ) returns p=8.46⋅10−13.
> wilcox.test(Height~factor(JPXB), data=LMK, alternative="greater")
Wilcoxon rank sum test with continuity correction
data: Height by factor(JPXB)
W = 33518, p-value = 8.455e-13
alternative hypothesis: true location shift is greater than 0
5) The Brunner-Munzel (doi: 10.1002/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U) test returns p=1.22⋅10−13.
> library(lawstat)
> brunner.munzel.test(LMK[LMK$JPXB==-1,]$Height, LMK[LMK$JPXB==0,]$Height, alternative = "greater", alpha=0.05)
Brunner-Munzel Test
data: LMK[LMK$JPXB == -1, ]$Height and LMK[LMK$JPXB == 0, ]$Height
Brunner-Munzel Test Statistic = -9.2965, df = 61.673, p-value = 1.215e-13
95 percent confidence interval:
0.1516498 0.2749568
sample estimates:
P(X<Y)+.5*P(X=Y)
0.2133033
6) The bootstrap-t-test as described by Wilcox (Introduction to robust estimation and hypothesis testing) returns a p close to 0.
> library(WRS2)
> yuenbt(Height~factor(JPXB), data=LMK, tr = 0, nboot=2000)
Call:
yuenbt(formula = Height ~ factor(JPXB), data = LMK, tr = 0, nboot = 2000)
Test statistic: 6.1419 (df = NA), p-value = 0
Trimmed mean difference: 20.12815
95 percent confidence interval:
13.57 26.6863
All tests provide significant evidence for rejecting the null-hypothesis that the two groups are in one way or another equivalent. (Or in the case of one-sided tests that the height of objects in the second group tends to be greater than in the first). Therefore, the question of choosing the right test to get the correct values, much discussed in other threads, is not urgent in my case. My question is how to report these findings in the most conventional way. I understand that the Yuen t-test, the Brunner-Munzel test, and the bootstrap-t remain somewhat esoteric and citing them in a study not particularly devoted to statistical methods may seem unjustified. I already cited the thread with arguments against the use of t-test on ranks (Equivalence of Mann-Whitney U-test and t-test on ranks). I also understand that there is no consensus whether the use of the Welch t-test on non-normally distributed data is justified by my sample size. Citing the Mann-Whitney test as the evidence of stochastic dominance looks like an unbeatable solution, but I truly suspect that this wording would sound at least puzzling to most readers in my field (archaeology). Maybe there exists a better way to convey this idea in more comprehensible terms? Thank you for your suggestions.
UPDATE 1 the objective of my tests: as suggested by subhash c. davar in his answer, I would like to make it clear that my primary objective is to check whether there is significant evidence to state that objects displaying a certain feature (coded as JPXB==-1) tend have greater height than objects without this feature (coded as JPXB==0). It would certainly also be nice to measure the distance (that is to say that objects of Group 1 tend to be at least X cm higher than those of Group 2 at CI=0.95), but I realise that I rather cannot with my skewed data.
UPDATE 2 the question in a nutshell: I have conducted 6 statistical tests on my data, which all provide significant evidence that the central tendencies of two independent populations are different. My question is which of these tests should I report (and in what way) to make my statement most convincing and to raise least questions among the readers in a research field little accustomed to statistics.
