# Dealing with very small samples with different shapes

I have two groups with very small sample sizes (just 6 obs per each group).It seems that There is a violation of normality and Homogeneity of variances (based on Plots).

Now I was wondering if I can use Mann-Whitney test to check the equality of medians although these two group donot have the same shape? Also as levene test doenot have the high power when we have small sample size, I though maybe I should compare the equality of variances based on rule of Thumb, But I am not sure whether the critical value for the rule of Thumb (equality of variances) is 3 or 9?

Any Advice would be highly appreciated.

This is the first analyte (More information: Both P values based on T-test and Mann-Whitney tests are not significant, T-test P value is 0.2 and Mann-Whitney test is 0.5)

This is the second analyte (More information: Both P-values based on T-test and Mann-Whitney tests are very small, around 0.002)

• The apparent "violation of normality" is an artifact of how your software chooses to draw reference lines on QQ plots. There is no evidence of non-normality here. The difference in variances is indeed large. The problem is that any rank-based test, like M-W, will have low power to detect even this much of a visible difference in medians. (I suspect that when you apply it you will not get a small p-value.) The lesson is that a difference has to be large and consistent if you wish to detect it with two groups of just half a dozen subjects each. – whuber Aug 29 '18 at 14:31
• Even though there is no proof of violation of normality, simply assuming normality in such small samples is usually not right If you do not have some knowledge about normality of this kind of data, a non parametric test like the Mann-Whitney test is in order - it will not be conclusive in the first analyte, but will be in the second. – Bernhard Aug 29 '18 at 14:57
• @ whuber, thanks for the comment, I just added a new figure to the question, I would be so thankful if you let me know what you think about the normality of the second figure too. Also If I want to check the equality of variances by using rule of Thumb (and not levene test) the critical value would be 3 or 9? – stat Aug 29 '18 at 14:59
• Given your boxplots those p-values do not come as a surprise. – mdewey Aug 29 '18 at 15:55
• The whole point to testing is to overcome visual and intuitive biases: to keep us from fooling ourselves (and each other). When dataset sizes are small and you "see" an effect, but that effect is not detected with an appropriate, correctly-computed test whose assumptions are not clearly and strongly violated by the data, then trust the test. In most cases you can confirm this advice through simulation. – whuber Aug 29 '18 at 16:57

A permutation test may be best for the first analyte. I don't have your exact data, so I will use the data below (roughly approximated from your graphs). I'm using the difference in group means as the metric. The P-value is 0.3. Among $m = 100,000$ permutations of group labels there were 94 distinct differences in means. [This P-value is essentially the same as the P-value of a 2-sided Welch t test.]

set.seed(829)
x1 = c(111,113,116,116,119,129);  x2 = c(105,112,125,125,138,142);  all=c(x1,x2)
gp = rep(1:2, each=6)
a1 = mean(x1);  a2 = mean(x2);  d.obs = a1 - a2
m = 10^5;  d.prm = numeric(m)
for(i in 1:m) {
prm = sample(gp)  # randomly permutes groups
d.prm[i] = mean(all[prm==1]) - mean(all[prm==2])  }
mean(abs(d.prm) >= abs(d.obs))
[1] 0.29665
length(unique(d.prm))
[1] 94

hdr = "Simulated Permutation Dist'n of Diff in Gp Means"
hist(d.prm, prob=T, col="skyblue2", main=hdr)
abline(v=c(-d.obs, d.obs), col="red", lwd=2, lty="dashed")


t.test(all~gp)

Welch Two Sample t-test

data:  all by gp
t = -1.1207, df = 6.8904, p-value = 0.3
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-22.337040   8.003706
sample estimates:
mean in group 1 mean in group 2
117.3333        124.5000

• The M-W is a permutation test. – whuber Aug 29 '18 at 16:58
• @whuber: Yes, but a particular kind of permutation test. As a test of differences in medians the M-W test has been questioned by a paper by Devine et al. (2018) "The Wilcoxon-Mann-Whitney procedure fails as a test of medians," in The American Statistician, Aug, p278-286. I have not had time to figure out exactly what the authors are deprecating and recommending. I suppose there may be letters to the editor on this one. – BruceET Aug 29 '18 at 17:11
• It is well-known that interpreting the test as a difference of medians becomes problematic when the distributions are asymmetric or differ in some way other than a mere shift of location. I haven't read the latest AmStat issue, so I'm not sure whether that article presents anything new or is just popularizing what is known--thank you for the reference. (+1) – whuber Aug 29 '18 at 17:16
• In any case, with as few as 6 observations in each group, I wonder about information loss reducing data to ranks. – BruceET Aug 29 '18 at 17:22
• @BruceET, thanks so much for the help. So it means as among 100000 permutations of group labels there were only 94 distinct differences in means, it means there is no evidence to reject the equality of means. Am I right? How I can calculate the P value based on this approach? – stat Aug 29 '18 at 20:03