How can a t-test be statistically significant if the mean difference is almost 0?

Question

I am trying to compare data from 2 populations to tell if the difference between the treatments is statistically significant. The data sets appear to be normally distributed with very little difference between the two sets. The average difference is 0.00017. I performed a paired t-test, expecting that I would fail to reject the null hypothesis of no difference between the means, however, my calculated t-value is much higher then my critical t-value.

Answer 1

To know if a difference is really large or small requires some measure of scale, the standard deviation is one measure of scale and is part of the t-test formula to account in part for that scale.

Consider if you are comparing the heights of 5 year olds to the heights of 20 year olds (humans, same geographic area, etc.). Intuition tells us that there is a practical difference there and if the heights are measured in inches or centimeters then the difference will look meaningful. But what if you convert the heights to kilometers? or light years? then the difference will be a very small number (but still different), but (barring round-off error) the t-test will give the same results whether the height is measured in inches, centimeters, or kilometers.

So a difference of 0.00017 may be huge depending on the scale of the measurements.

Answer 2

I see no reason to believe you did something wrong just because the test was significant, even if the mean difference is very small. In a paired t-test, the significance will be driven by three things:

1. the magnitude of the mean difference
2. the amount of data you have
3. the standard deviation of the differences

Admittedly, your mean difference is very, very small. On the other hand, you do have a fair amount of data (N=335). The last factor is the standard deviation of the differences. I don't know what that is, but since you got a significant result, it is safe to assume it is small enough to overcome the small mean difference with the amount of data you have. For the sake of building an intuition, imagine that the paired difference for every observation in your study were 0.00017, then the standard deviation of the differences would be 0. Surely, it would be reasonable to conclude that the treatment led to a reduction (albeit a tiny one).

As @whuber notes in the comments below, it is worth pointing out that while 0.00017 seems like a very small number qua number, it isn't necessarily small in meaningful terms. To know that, we would need to know several things, firstly what the units are. If the units are very large (e.g., years, kilometers, etc.), what appears to be small could be meaningfully large, whereas if the units are small (e.g., seconds, centimeters, etc.), this difference seems even smaller. Second, even a small change can be important: imagine some kind of treatment (e.g., vaccine) that was very cheap, easy to administer to the whole populace, and had no side effects. It may well be worth doing even if it saved only a very few lives.

Answer 3

If your critical $t$ is less than what you calculated, and assuming the test was appropriate for your particular kind of data (an important "if"), it seems your difference is statistically significant in the sense of unlikely to emerge at least as large in another, similar pair of samples selected randomly from the same populations if the null hypothesis of no difference is literally true of those populations. A significant $t$ in the appropriate context generally means that your observed difference is too reliably non-zero to support the null hypothesis that the data are not "any different at all". Even a difference of $\frac{17}{100,000}$ can be statistically significant from zero if every observed difference is between .00015–.00020. Observe!

pop1=rep(15:20* .00001, 56);pop2=rep(0,336) #Some fake samples of sample size = 336
t.test(pop1,pop2,paired=T)                #Paired t-test with the following output...

$$t_{(335)}=187.55,p<2.2\times10^{-16}$$

Because these samples are very consistently different, the difference achieves statistical significance, even though they are of smaller scale than many of us are used to seeing in mundane, everyday numbers. In fact, you can scale down the data as much as you like by tacking as many zeros as your calculations can handle onto to the front of .00001 in my first line of R code. This will scale down the standard deviation of the differences as well; i.e., your differences will remain just as consistent, your $t$ will remain exactly the same, and so will its significance.

Maybe you'd be more interested in practical significance than in this literal sense of null hypothesis significance testing. Practical significance will depend much more on the meaning of your data in context than on statistical significance; it is not a purely statistical matter. I cited a useful example of this principle in an answer to a popular question here, Accommodating entrenched views of p-values:

One cannot conclude by size alone that an $r=.03$ is necessarily unimportant if it might pertain to a matter of life and death ^{[(Rosenthal, Rubin, & Rosnow, 2000)]}.

This "matter of life and death" was the effect size of aspirin on heart attacks, basically – a powerful example of numerically small, much less consistent differences with practically important meaning. Many other questions with solid answers from which you might benefit deserve links here, including:

• Why is "statistically significant" not enough?
• practical vs statistical significance
• Practical significance, especially with percents: "standard" measure and threshold

Reference

Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. Cambridge University Press.

Answer 4

Here is an example in R that shows the theoretical concepts in action. 10,000 trials of flipping a coin 10,000 times that has a probability of heads of .0001 compared to 10,000 trials of flipping a coin 10,000 times that has a probability of heads of .00011

t.test(rbinom(10000, 10000, .0001), rbinom(10000, 10000, .00011))

t = -8.0299, df = 19886.35, p-value = 1.03e-15 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -0.14493747 -0.08806253 sample estimates: mean of x mean of y 0.9898 1.1063

The difference in the mean is relatively closed to 0 in terms of human perception, however it is very statistically different than 0.