# How can a t-test be statistically significant if the mean difference is almost 0?

I am trying to compare data from 2 populations to tell if the difference between the treatments is statistically significant. The data sets appear to be normally distributed with very little difference between the two sets. The average difference is 0.00017. I performed a paired t-test, expecting that I would fail to reject the null hypothesis of no difference between the means, however, my calculated t-value is much higher then my critical t-value.

• What do you want suggestions about? What are your N's? – gung - Reinstate Monica Feb 25 '14 at 20:11
• hi, i'm just not really sure how to proceed, if i did something wrong to begin with seeing as how the data doesn't seem to be any different at all. Both groups have 335 observations – Kscicc26 Feb 25 '14 at 20:17
• The standard error of the difference in means is also a function of the standard deviations and of the sample sizes. All these pieces would need to be in your question before any surprise could be registered. – Glen_b -Reinstate Monica Feb 25 '14 at 20:34
• Every difference is "almost 0"! If the outcome variable is weight gained by people and it is measured in pounds, then 0.00017 is small indeed, but if it is measured in millions of pounds then 0.00017 is enormous. This question therefore has no meaning until a context--what is being measured in the response--and a unit of measurement are provided. – whuber Feb 25 '14 at 21:02
• Statistical significance doesn't mean "significance" in the wider English sense of significance. – david25272 Feb 26 '14 at 0:14

I see no reason to believe you did something wrong just because the test was significant, even if the mean difference is very small. In a paired t-test, the significance will be driven by three things:

1. the magnitude of the mean difference
2. the amount of data you have
3. the standard deviation of the differences

Admittedly, your mean difference is very, very small. On the other hand, you do have a fair amount of data (N=335). The last factor is the standard deviation of the differences. I don't know what that is, but since you got a significant result, it is safe to assume it is small enough to overcome the small mean difference with the amount of data you have. For the sake of building an intuition, imagine that the paired difference for every observation in your study were 0.00017, then the standard deviation of the differences would be 0. Surely, it would be reasonable to conclude that the treatment led to a reduction (albeit a tiny one).

As @whuber notes in the comments below, it is worth pointing out that while 0.00017 seems like a very small number qua number, it isn't necessarily small in meaningful terms. To know that, we would need to know several things, firstly what the units are. If the units are very large (e.g., years, kilometers, etc.), what appears to be small could be meaningfully large, whereas if the units are small (e.g., seconds, centimeters, etc.), this difference seems even smaller. Second, even a small change can be important: imagine some kind of treatment (e.g., vaccine) that was very cheap, easy to administer to the whole populace, and had no side effects. It may well be worth doing even if it saved only a very few lives.

• thank you for the response! I'm not too versed in statistics, so i was just taken aback when i didn't get the answer i was expecting to get. the standard error of the differences between the means is: 7.36764E-05. I'm not sure what the relevance of that is, but i'm sure you do haha. again thanks for your help – Kscicc26 Feb 25 '14 at 20:46
• You're welcome, @Kscicc26. The standard error of the differences, & the standard deviation of the differences are not the same thing. (Tragically, they sound like they ought to be.) The SD tells you how much your differences vary, whereas the SE tells you how much estimates of the mean difference would vary if you ran your study over & over & over again. It may help you to read my description of SEs here. – gung - Reinstate Monica Feb 25 '14 at 20:50
• i will check that out and get back on this thread in the morning! – Kscicc26 Feb 25 '14 at 20:51
• This mean difference is neither small nor large: you simply have no basis for assessing its size. – whuber Feb 25 '14 at 21:03
• @whuber, that's a good point--I don't know what these numbers refer to. But the OP presumably does & thinks it is very small. I'm going with that information. – gung - Reinstate Monica Feb 25 '14 at 21:09

To know if a difference is really large or small requires some measure of scale, the standard deviation is one measure of scale and is part of the t-test formula to account in part for that scale.

Consider if you are comparing the heights of 5 year olds to the heights of 20 year olds (humans, same geographic area, etc.). Intuition tells us that there is a practical difference there and if the heights are measured in inches or centimeters then the difference will look meaningful. But what if you convert the heights to kilometers? or light years? then the difference will be a very small number (but still different), but (barring round-off error) the t-test will give the same results whether the height is measured in inches, centimeters, or kilometers.

So a difference of 0.00017 may be huge depending on the scale of the measurements.

If your critical $t$ is less than what you calculated, and assuming the test was appropriate for your particular kind of data (an important "if"), it seems your difference is statistically significant in the sense of unlikely to emerge at least as large in another, similar pair of samples selected randomly from the same populations if the null hypothesis of no difference is literally true of those populations. A significant $t$ in the appropriate context generally means that your observed difference is too reliably non-zero to support the null hypothesis that the data are not "any different at all". Even a difference of $\frac{17}{100,000}$ can be statistically significant from zero if every observed difference is between .00015–.00020. Observe!

pop1=rep(15:20* .00001, 56);pop2=rep(0,336) #Some fake samples of sample size = 336
t.test(pop1,pop2,paired=T)                #Paired t-test with the following output...


$$t_{(335)}=187.55,p<2.2\times10^{-16}$$

Because these samples are very consistently different, the difference achieves statistical significance, even though they are of smaller scale than many of us are used to seeing in mundane, everyday numbers. In fact, you can scale down the data as much as you like by tacking as many zeros as your calculations can handle onto to the front of .00001 in my first line of R code. This will scale down the standard deviation of the differences as well; i.e., your differences will remain just as consistent, your $t$ will remain exactly the same, and so will its significance.

Maybe you'd be more interested in practical significance than in this literal sense of null hypothesis significance testing. Practical significance will depend much more on the meaning of your data in context than on statistical significance; it is not a purely statistical matter. I cited a useful example of this principle in an answer to a popular question here, Accommodating entrenched views of p-values:

One cannot conclude by size alone that an $r=.03$ is necessarily unimportant if it might pertain to a matter of life and death [(Rosenthal, Rubin, & Rosnow, 2000)].

This "matter of life and death" was the effect size of aspirin on heart attacks, basically – a powerful example of numerically small, much less consistent differences with practically important meaning. Many other questions with solid answers from which you might benefit deserve links here, including:

Reference

Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. Cambridge University Press.

Here is an example in R that shows the theoretical concepts in action. 10,000 trials of flipping a coin 10,000 times that has a probability of heads of .0001 compared to 10,000 trials of flipping a coin 10,000 times that has a probability of heads of .00011

t.test(rbinom(10000, 10000, .0001), rbinom(10000, 10000, .00011))

t = -8.0299, df = 19886.35, p-value = 1.03e-15 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -0.14493747 -0.08806253 sample estimates: mean of x mean of y 0.9898 1.1063

The difference in the mean is relatively closed to 0 in terms of human perception, however it is very statistically different than 0.