4
$\begingroup$

To begin with, I must admit, I am completely new to statistics. There are many discussions on CrossValidated on the optimal ways to compare the central tendency of two independent populations, which violate normality and homogeneity assumptions. Among the most useful threads I may cite the following: Mann Whitney test with unequal variances; Unequal variances t-test or U Mann-Whitney test?; Reporting Mann-Whitney U-Test without homogenity of variance; How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples; Equivalence of Mann-Whitney U-test and t-test on ranks; and Bootstrap vs. permutation hypotheis testing . Following the arguments and references put forward in these treads (and cited in the introductive works, available to me), I fell under the impression that there is no single approach for solving the problem, but rather different tests demonstrate different levels of power under different conditions (as is argued by Fagerland and Sandvik 2009 (doi 10.1016/j.cct.2009.06.007)) and suit different purposes (Greg Snow in Bootstrap vs. permutation hypotheis testing). I am an archaeologist. I am testing the hypothesis whether objects of a certain type displaying a certain property (boolean factor JPXB in the examples below) tend to be higher than objects of the same type without this property. Thus, I have two groups in my population (which is a possibly exhaustive catalogue of objects of this type known today). The heights in both groups violate normality (transformations do not help) and their distributions are different.
Group 1: n=54, median=54.00cm, mean=61.19cm, range=108cm, std.dev=23.76cm, Shapiro-Wilk W=0.88 with p=5.69⋅10−5, skewness=1.22;
Group 2: n=789, median=40.00cm, mean=41.06cm, range=112cm, std.dev=14.90cm, Shapiro-Wilk W=0.94 with p=1.2⋅10−16, skewness=1.09.
That both groups deviate from normality is evident from QQ-plots, and that the groups are different is confirmed by Levene test F=10.66, p=0.0011. The different tests proposed in the literature and in discussions on CrossValidated for such data return the same results.

1) The one-sided Welch t-test, which is claimed by some to be reasonably robust to perform well on large enough samples violating normality (see, particularly the CrossValidated threads What normality assumptions are required for an unpaired t-test? And when are they met?, How robust is the independent samples t-test when the distributions of the samples are non-normal? and How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples ), which should be my case with n=54 and 789, returns p=4.49⋅10−8.

>t.test(Height~factor(JPXB), data=LMK, alternative="greater")
    Welch Two Sample t-test  
data:  Height by factor(JPXB)    
t = 6.1419, df = 55.888, p-value = 4.492e-08  
alternative hypothesis: true difference in means is greater than 0  
95 percent confidence interval:
 14.64681      Inf
sample estimates:
mean in group -1  mean in group 0 
    61.18519         41.05703 

2) The one-sided Welch t-test of ranks (referred to as the Welch U test in Fagerland and Sandvik 2009 (doi: 10.1016/j.cct.2009.06.007) and considered inappropriate in the replies to the thread Equivalence of Mann-Whitney U-test and t-test on ranks), returns p=2.05⋅10−13.

> t.test(rank(Height)~factor(JPXB), data=LMK, alternative="greater")
    Welch Two Sample t-test
data:  rank(Height) by factor(JPXB)
t = 9.0288, df = 65.557, p-value = 2.048e-13
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
 197.0243      Inf
sample estimates:
mean in group -1  mean in group 0 
        648.2037         406.5184 

3) The Yuen t-test with 20% trimming returns p=2.72⋅10−6.

> library(PairedData)
> yuen.t.test(Height~factor(JPXB), data=LMK, tr = 0.2, alternative="greater")

    Two-sample Yuen test, trim=0.2

data:  Height by factor(JPXB)
t = 5.3531, df = 35.169, p-value = 2.716e-06
alternative hypothesis: true difference in trimmed means is greater than 0
95 percent confidence interval:
 11.30603      Inf
sample estimates:
trimmed mean in group -1  trimmed mean in group 0 
                56.38235                 39.86316 

4) The Mann–Whitney test (which should in the case of non-homogeneous samples provide the evidence of stochastic dominance, and not measure the location shift, so Reporting Mann-Whitney U-Test without homogenity of variance ) returns p=8.46⋅10−13.

> wilcox.test(Height~factor(JPXB), data=LMK, alternative="greater")

    Wilcoxon rank sum test with continuity correction

data:  Height by factor(JPXB)
W = 33518, p-value = 8.455e-13
alternative hypothesis: true location shift is greater than 0

5) The Brunner-Munzel (doi: 10.1002/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U) test returns p=1.22⋅10−13.

> library(lawstat)
> brunner.munzel.test(LMK[LMK$JPXB==-1,]$Height, LMK[LMK$JPXB==0,]$Height, alternative = "greater", alpha=0.05)
    Brunner-Munzel Test
data:  LMK[LMK$JPXB == -1, ]$Height and LMK[LMK$JPXB == 0, ]$Height
Brunner-Munzel Test Statistic = -9.2965, df = 61.673, p-value = 1.215e-13
95 percent confidence interval:
 0.1516498 0.2749568
sample estimates:
P(X<Y)+.5*P(X=Y) 
       0.2133033 

6) The bootstrap-t-test as described by Wilcox (Introduction to robust estimation and hypothesis testing) returns a p close to 0.

> library(WRS2)
> yuenbt(Height~factor(JPXB), data=LMK, tr = 0, nboot=2000)
Call:
yuenbt(formula = Height ~ factor(JPXB), data = LMK, tr = 0, nboot = 2000)
Test statistic: 6.1419 (df = NA), p-value = 0

Trimmed mean difference:  20.12815 
95 percent confidence interval:
13.57     26.6863 

All tests provide significant evidence for rejecting the null-hypothesis that the two groups are in one way or another equivalent. (Or in the case of one-sided tests that the height of objects in the second group tends to be greater than in the first). Therefore, the question of choosing the right test to get the correct values, much discussed in other threads, is not urgent in my case. My question is how to report these findings in the most conventional way. I understand that the Yuen t-test, the Brunner-Munzel test, and the bootstrap-t remain somewhat esoteric and citing them in a study not particularly devoted to statistical methods may seem unjustified. I already cited the thread with arguments against the use of t-test on ranks (Equivalence of Mann-Whitney U-test and t-test on ranks). I also understand that there is no consensus whether the use of the Welch t-test on non-normally distributed data is justified by my sample size. Citing the Mann-Whitney test as the evidence of stochastic dominance looks like an unbeatable solution, but I truly suspect that this wording would sound at least puzzling to most readers in my field (archaeology). Maybe there exists a better way to convey this idea in more comprehensible terms? Thank you for your suggestions.

UPDATE 1 the objective of my tests: as suggested by subhash c. davar in his answer, I would like to make it clear that my primary objective is to check whether there is significant evidence to state that objects displaying a certain feature (coded as JPXB==-1) tend have greater height than objects without this feature (coded as JPXB==0). It would certainly also be nice to measure the distance (that is to say that objects of Group 1 tend to be at least X cm higher than those of Group 2 at CI=0.95), but I realise that I rather cannot with my skewed data.

UPDATE 2 the question in a nutshell: I have conducted 6 statistical tests on my data, which all provide significant evidence that the central tendencies of two independent populations are different. My question is which of these tests should I report (and in what way) to make my statement most convincing and to raise least questions among the readers in a research field little accustomed to statistics.

$\endgroup$
  • 1
    $\begingroup$ Could you please make the question clearer? I see you put much effort in this question, but as it currently stands the wall of text is quite daunting. $\endgroup$ – Firebug Oct 3 '16 at 13:30
1
$\begingroup$

It seems like a prudent path would be to go with well known methodology, given your audience is non-statisticians. You have non-normal distributions and good sample size. You wish to report a difference in central tendency and confidence around an estimate of central tendency; you're particularly interested in an estimate of central tendency your audience can quickly understand. It seems like a Mann-Whitney U test and bootstrap confidence intervals would be what you're looking for. I'd also recommend a box plot with the two groups; graphs are worth a thousand tests. That should cause your audience to immediately key in on the difference you're seeing.

Mann-Whitney U tests for differences in the rank sum between the two groups. It would detect if the ranks of one group are generally higher or lower than those of another group. Since you have heteroskedasticy, the test will strictly detect differences only in ordering of the groups, not just the median. The $p$-value is the probability the mean ranks of the two groups would be as far apart as they are given random shuffling of the values.

To get a confidence interval, you likely have sufficient sample size to bootstrap the estimate of your choice. Calculate the difference in median between the groups; resample and repeat several times (1,000 to 10,000 is usually good). If you like two-sided 95% confidence, you can report the 0.025, 0.5, and 0.975 quantiles from your repeated differences in median as the lower confidence limit, estimate, and upper confidence limit, respectively.

As a small aside, I saw in your above examples you used one-sided confidence. Best practice is to determine that you will use one-sided tests prior to collecting any data, so unless you determined that before looking at the data, sticking with two-sided methodology would be best.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.