# t-tests give weird results for samples with very large differences between means

I have the following dataset mices.dat:

Number Year
80 2000
247 2000
116 2000
130 2000
1291 2020
2265 2020
495 2020


Here are some descriptive stats for two groups:

$2000 Min. 1st Qu. Median Mean 3rd Qu. Max. 80.0 107.0 123.0 143.2 159.2 247.0$2020
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
495     893    1291    1350    1778    2265


As you can see interquartile ranges doesn't overlap at all and absolute difference in means is big too. I've even plotted the following boxplot: I'm trying to perform t-test to find if there are any significant differences between means in two groups. I've done normality test (and it looks good) followed by F-test (which is good too). But both unpaired two samples t-test and Mann-Whitney U-test gives me weird result that means are equal! The whole R script is here:

# load and analyze data
library(dplyr)
df <- mutate(df, Number = as.numeric(Number), Year = factor(Year, levels = unique(Year)))
y2000 <- df[which(df$$Year == 2000), ] y2020 <- df[which(df$$Year == 2020), ]

tapply(df$$Number, df$$Year, summary) # means are very different
boxplot(Number ~ Year, data = df, ylab = "Number", xlab = "Year") # boxplot visually confirms it

# perform some tests
library(ggpubr)
ggqqplot(y2000$$Number) shapiro.test(y2000$$Number) # p-value = 0.3152 => normal distribution
ggqqplot(y2020$$Number) shapiro.test(y2020$$Number) # p-value = 0.8892 => normal distribution

var.test(y2000$$Number, y2020$$Number, alternative = "two.sided") # p-value = 0.001964 => variances differ

t.test(y2000$$Number, y2020$$Number, alternative = "two.sided", var.equal = FALSE) # p-value = 0.1417 => not differs; recheck with Mann-Whitney U-test below
wilcox.test(y2000$$Number, y2020$$Number, alternative = "two.sided") # p-value = 0.05714 => not differs


I don't know why I'm getting this result. I suspect that this can be related to small sample sizes. So my question is: am I doing something wrong in that situation? If so, what is the proper way to present results? What kind of tests should I use to get some meaningful results?

It's obvious that these groups are different even from looking at summary and boxplot, but formal testing says opposite...

• Did you make a typo in the data values you posted?
– Dave
Jun 4 '20 at 21:53
• The problem lies with the boxplots: they are drawn too nicely and do not reveal the paucity of the underlying data.
– whuber
Jun 4 '20 at 21:55
• @Dave no, I've rechecked it several times Jun 4 '20 at 22:08
• In what sense do you regard the F-test as "good"? What do you think it tells you? Jun 5 '20 at 5:34
• @Glen_b-ReinstateMonica I got value around 0.002 which is definitely less than 0.05 so I made a conclusion that two samples have different variances and should state unequality of vars in t-test explicitly Jun 5 '20 at 8:13

The t-test is right. Here's a dotplot, which gives a different impression from the boxplot, because it highlights the tiny sample size The true mean of first group must be close to 143; the second group has a large variance and only three observations, so you can't rule out its mean also being close to 143, or even lower.

(The Shapiro test is misleading because it has essentially no power with three observations: it is never valid to conclude the null hypothesis is correct just because the $$p$$-value is large)

If you take a log transformation of Number you get

> t.test(log(Number)~Year,data=mices)

Welch Two Sample t-test

data:  log(Number) by Year
t = -4.2877, df = 3.1108, p-value = 0.02166
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-3.7190908 -0.5866796
sample estimates:
mean in group 2000 mean in group 2020
4.878135           7.031020


There is modest evidence that the means of log(Number) differ, because you're now explaining the higher variance in the second group as due to a variance:mean relationship.

Since monotone transformations do not affect the Mann-Whitney-Wilcoxon test, you'd expect there to be some transformation that makes the t-test agree fairly well, and log() seems to be it.

• +1. The problem with applying a transformation is that it takes out another degree of freedom, making it impossible to establish any significant difference. It's nevertheless useful as an exploratory tool.
– whuber
Jun 4 '20 at 21:56
• It does if you do it after looking at the data, but with a variance-mean relationship this strong, you'd hope it was known a priori. Jun 4 '20 at 22:02
• I agree--that's legitimate.
– whuber
Jun 4 '20 at 22:03
• Thank you a lot for detailed explanation. Nevertheless, even if mean value for the first group is around 143, all values in second sample are much greater even than maximum of the first. So in any way means should differ a lot, as it seems to me Jun 4 '20 at 22:13
• No, that's not correct. It's quite possible with a sample size of three to get all three observations well above the mean. With another couple of observations you'd be correct. Jun 4 '20 at 22:16