# Why the p-value of t.test() is not statistically significant when mean values look really different

I am trying to find out whether there is a significant difference in the mean value of a biomarker between two groups. I am using t.test in R. The mean(SD) values are 1142(1079) and 864(922)in the groups. But the p-values of the test shows the difference is not statistically significant. Can someone please help me? I am sharing the dput of the dataframe below.

structure(list(ANGPTL7 = c(2.5, 205, 885, 1915, 835, 1685, 625,
1615, 84.9999999999999, 1175, 2695, 235, 1025, 2.5, 2915, 825,
255, 1085, 1815, 2.5, 205, 985, 2.5, 705, 435, 555, 2045, 135,
15, 975, 2285, 1905, 515, 74.9999999999999, 25, 815, 1075, 2.5,
1115, 3115, 64.9999999999999, 64.9999999999999, 325, 595, 285,
2.5, 2.5, 345, 5.00000000000001, 215, 3465, 555, 855, 3745, 25,
305, 2.5, 2.5, 15, 115, 565, 94.9999999999999, 1005, 575, 405,
2.5, 1855, 1795, 145, 2555, 1705, 74.9999999999999, 735, 375,
2.5, 475, 1675, 1105, 345, 385, 3195, 115, 1475, 205, 545, 1265,
485, 1135, 2595, 3305, 305, 575, 1415, 2925, 3125, 2795, 3125,
1775, 1125, 15, 1695, 1225, 1625, 3175, 3185, 1445, 3065, 785,
855, 1115, 145, 595, 435, 185, 345, 2455, 1885), OSA_status = c("Non-OSA",
"OSA", "Non-OSA", "OSA", "Non-OSA", "OSA", "Non-OSA", "OSA",
"Non-OSA", "Non-OSA", "Non-OSA", "OSA", "OSA", "OSA", "OSA",
"OSA", "Non-OSA", "Non-OSA", "OSA", "OSA", "Non-OSA", "Non-OSA",
"OSA", "Non-OSA", "OSA", "OSA", "OSA", "OSA", "Non-OSA", "Non-OSA",
"Non-OSA", "OSA", "OSA", "OSA", "OSA", "Non-OSA", "OSA", "OSA",
"OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA",
"OSA", "Non-OSA", "Non-OSA", "Non-OSA", "OSA", "Non-OSA", "OSA",
"OSA", "OSA", "OSA", "Non-OSA", "Non-OSA", "OSA", "OSA", "Non-OSA",
"OSA", "OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA",
"OSA", "OSA", "OSA", "OSA", "OSA", "OSA", "OSA", "OSA", "OSA",
"OSA", "OSA", "Non-OSA", "OSA", "Non-OSA", "OSA", "Non-OSA",
"Non-OSA", "OSA", "Non-OSA", "OSA", "Non-OSA", "OSA", "OSA",
"OSA", "OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA",
"OSA", "Non-OSA", "Non-OSA", "Non-OSA", "OSA", "Non-OSA", "OSA",
"OSA", "OSA", "OSA", "Non-OSA", "Non-OSA", "OSA", "OSA", "OSA",
"OSA", "OSA", "OSA", "Non-OSA", "OSA", "OSA")), row.names = c(NA,
-117L), class = c("tbl_df", "tbl", "data.frame"))



EDIT This data example is great for R users but a small or large pain for everyone else. This format may or may not be easier, depending.

id  ANGPTL7 OSA_status
1       2.5    Non-OSA
2     205.0        OSA
3     885.0    Non-OSA
4    1915.0        OSA
5     835.0    Non-OSA
6    1685.0        OSA
7     625.0    Non-OSA
8    1615.0        OSA
9      85.0    Non-OSA
10   1175.0    Non-OSA
11   2695.0    Non-OSA
12    235.0        OSA
13   1025.0        OSA
14      2.5        OSA
15   2915.0        OSA
16    825.0        OSA
17    255.0    Non-OSA
18   1085.0    Non-OSA
19   1815.0        OSA
20      2.5        OSA
21    205.0    Non-OSA
22    985.0    Non-OSA
23      2.5        OSA
24    705.0    Non-OSA
25    435.0        OSA
26    555.0        OSA
27   2045.0        OSA
28    135.0        OSA
29     15.0    Non-OSA
30    975.0    Non-OSA
31   2285.0    Non-OSA
32   1905.0        OSA
33    515.0        OSA
34     75.0        OSA
35     25.0        OSA
36    815.0    Non-OSA
37   1075.0        OSA
38      2.5        OSA
39   1115.0        OSA
40   3115.0    Non-OSA
41     65.0    Non-OSA
42     65.0    Non-OSA
43    325.0    Non-OSA
44    595.0    Non-OSA
45    285.0        OSA
46      2.5    Non-OSA
47      2.5    Non-OSA
48    345.0    Non-OSA
49      5.0        OSA
50    215.0    Non-OSA
51   3465.0        OSA
52    555.0        OSA
53    855.0        OSA
54   3745.0        OSA
55     25.0    Non-OSA
56    305.0    Non-OSA
57      2.5        OSA
58      2.5        OSA
59     15.0    Non-OSA
60    115.0        OSA
61    565.0        OSA
62     95.0    Non-OSA
63   1005.0    Non-OSA
64    575.0    Non-OSA
65    405.0    Non-OSA
66      2.5    Non-OSA
67   1855.0        OSA
68   1795.0        OSA
69    145.0        OSA
70   2555.0        OSA
71   1705.0        OSA
72     75.0        OSA
73    735.0        OSA
74    375.0        OSA
75      2.5        OSA
76    475.0        OSA
77   1675.0        OSA
78   1105.0    Non-OSA
79    345.0        OSA
80    385.0    Non-OSA
81   3195.0        OSA
82    115.0    Non-OSA
83   1475.0    Non-OSA
84    205.0        OSA
85    545.0    Non-OSA
86   1265.0        OSA
87    485.0    Non-OSA
88   1135.0        OSA
89   2595.0        OSA
90   3305.0        OSA
91    305.0        OSA
92    575.0    Non-OSA
93   1415.0    Non-OSA
94   2925.0    Non-OSA
95   3125.0    Non-OSA
96   2795.0    Non-OSA
97   3125.0        OSA
98   1775.0    Non-OSA
99   1125.0    Non-OSA
100    15.0    Non-OSA
101  1695.0        OSA
102  1225.0    Non-OSA
103  1625.0        OSA
104  3175.0        OSA
105  3185.0        OSA
106  1445.0        OSA
107  3065.0    Non-OSA
108   785.0    Non-OSA
109   855.0        OSA
110  1115.0        OSA
111   145.0        OSA
112   595.0        OSA
113   435.0        OSA
114   185.0        OSA
115   345.0    Non-OSA
116  2455.0        OSA
117  1885.0        OSA


I agree with @pikachu that the standard deviations are too large compared with the difference between means for a t test to find a significant difference.

Thank you for posting your data. It is always a good idea to take a look at some graphic displays of the data before doing formal tests.

Stripcharts of observations in the two groups do not show a meaningful difference in locations relative to the variability of the samples.

stripchart(ANGPTL7 ~OSA_status, pch="|", ylim=c(.5,2.5))


Here are boxplots of the two groups. The 'notches' in the the sides of the boxes are nonparametric confidence intervals, calibrated so that overlapping notches tend to indicate no significant difference in location.

boxplot(ANGPTL7 ~ OSA_status, notch=T,
col="skyblue2", horizontal=T)


Even with sample sizes as large as these, I would be reluctant to do a two-sample t test on account of the marked skewness of the data. I would do a nonparametric two-sample Wilcoxon rank sum test (which also shows no significant difference).

wilcox.test(ANGPTL7 ~ OSA_status)

Wilcoxon rank sum test with continuity correction

data:  ANGPTL7 by OSA_status
W = 1456.5, p-value = 0.2139
alternative hypothesis: true location shift is not equal to 0

• (+1) The displays certainly help greatly, but leave scope for second-order grumbling about whether other displays would work even better. The rules behind the box plot -- what and what is not plotted in the tails depend on what is less than or more than 1.5 IQR from the nearer quartile -- lead to different decisions in the upper tail in a way that doesn't help very much. Also, although it is likely to be unimportant, there is fine structure in the data that the box plots can't even hint at. Commented Apr 7, 2021 at 11:40
• You can have distributions with higher mean but lower rank in a pairwise comparison. Wilcoxon rank test is not always/necessarily a test for equality of means. It is more precisely a test for equality of distributions. (I agree if you would argue that the OP is probably not really looking for ”trying to find out whether there is a significant difference in the mean value". My comment is just a sidenote, warning for the potentially wrong interpretation of Wilcoxon rank test as a test for equality of means.) Commented Apr 7, 2021 at 20:43
• Perhaps like @SextusEmpiricus I want to underline that it's a really big jump between t tests for which normal distributions are ideal and Wilcoxon-Mann-Whitney tests which test less than people hope.. The example of Poisson distributions shows that means can be natural quantities even for possibly very skewed distributions, so it is just a case of finding the best machinery for the problem. The example is pertinent because the outcome here appears to be some kind of count. Commented Apr 7, 2021 at 21:35
• Thanks for comments by @SextusEmpiricus and comments and answer by Nick Cox. After various experiments of my own, I decided to focus on differences in shape as well as differences in mean. Commented Apr 8, 2021 at 5:01
• @user257566. Don't see how you could read my accompanying explanations and come to that conclusion. // I almost always upvote other helpful answers in parallel with mine, but your Answer assumes normality from the start and without question, whereas OP's data seem aggressively non-normal. At the very least you need to address grossly unequal variances and caution against pooled t test. // We obviously have different views on these data. Your are, of course, entitled to yours. Commented Apr 10, 2021 at 17:04

When you consider the difference between means you have to use a different unit than the simple absolute difference. Take into account that you are measuring the difference in means produced by two random sources. Theose random sources (whose outcomes are your two samples) contains variability. It is this variability which should be used to compare the difference in means. The standard deviations are measured in the same units as the difference so you should see the difference in terms of standard deviations.

Now you see that the difference in means $$|1142-864|$$ is much lower than any of the sample standard deviations by far. If you approximate with a normal distribution than you will see that the difference in means is about one third of a standard deviation (whatever one can be used) and you can see that there are a lot of chances that the difference in mean to be produced simply by random errors. Use as an approximation that in normal distribution with one standard deviation on the left of mean and on the right you will get aproximately 60% of your data, and with 2 standard deviations you will cover around 95% of your data. The mean difference is far less than any standard deviations so you should consider them equal.

To illustrate the idea of using the mean difference and variation for comparison you can take a look at this image When you compare the means taking into account the variance you see that the two distributions are not very different, their overlap is large.

When you work out the t test for difference the distribution for the difference in means looks like the following: The vertical bar is where your mean difference is in terms of standard deviations and the distribution displays how it will variate. It should be clar that the value of the standard deviation scaled difference in mean is not an exceptional value and as a consequence it cannot produce a small p-value.

With respect to plotting the data, I'd like to point the R package ggbeeswarm. In this case I think it is better than a boxplot or violin plot. The horizontal segments are the quantiles at 5%, 50%, 95%; dat is the data from the OP

library(ggplot2)
library(ggbeeswarm)
library(data.table)

dat <- as.data.table(dat)
dat[, OSA_status := as.factor(OSA_status)]
qq <- dat[, list(quantile= quantile(ANGPTL7, p= c(0.05, 0.5, 0.95))), by= OSA_status]

gg <- ggplot(data= dat, aes(x= OSA_status, y= ANGPTL7)) +
geom_quasirandom(width= 0.25) +
geom_segment(data= qq, aes(y= quantile, yend= quantile, x= as.numeric(OSA_status)-0.1, xend= as.numeric(OSA_status)+0.1), colour= 'blue') +
theme_light()

• In spirit this plot resembles mine. The jittering it uses is sometimes exactly what is best but at other times just adds gratuitous noise. Commented Apr 8, 2021 at 17:37

You have too high standard deviation (uncertainty) and with selected probability (I assume $$\alpha = 0.05$$ that is by default) the confidence intervals are overlapping, thus no statistically significant difference between means.

EDIT: CI of one mean covers the estimate of the second mean, thus they are not significantly different. Thanks for correction from comments.

• Overlapping confidence intervals does not preclude a significant difference.
– Dave
Commented Apr 6, 2021 at 19:21
• @Dave is correct, a confidence interval overlapping a point means the parameter is not significantly different from that value. But a confidence interval overlapping another confidence interval can still be significantly different, since it's unlikely that the second value is actually at that extreme of its CI. Two parameters could both be insignificantly different from 0 (both CIs overlap 0), but still be significantly different from one another. Commented Apr 6, 2021 at 19:34
• Thanks for the remarks, you are right, I will incorporate them to my answer so that it does not confuse other people. Commented Apr 7, 2021 at 7:13
• See for more about the confidence intervals and overlap: Why is mean ± 2*SEM (95% confidence interval) overlapping, but the p-value is 0.05?. Commented Apr 10, 2021 at 19:18

Excellent points in other answers. I went a little further in looking at the data. The common practice of firing up box plots when the question is about means is a good idea because a good box plot will give you a good overview of the data, but it is a little indirect whenever -- as is a typical default in software -- the means aren't shown at all, but have to be guessed at, or inserted mentally from another calculation, or indeed added whenever the software allows. I started out with something a little like @BruceET's display, considered a logarithmic transformation and then retreated because that went too far. A square root scale seemed to work better for visualization.

This display is a hybrid quantile and box plot using square root scale. Each box plot is minimal and shows only minimum, maximum, median and quartiles, for all of which it is true in principle that summary(root()) $$=$$ root(summary()). The box plot is minimal because alongside each a quantile plot is shown with all the observed values in order. The horizontal reference line is at the mean of the square roots, which is close to the median in each case.

Fine structure is revealed by the quantile plot that the box plot conceals, and an easy tabulation shows that all values are multiples of 5, except that there is a minor mode at the lowest reported value, 2.5. That leads me to guess at a detection limit problem, such that any observed zero has been fudged, nudged or kludged to half the smallest non-zero value of 5. (I take it that the data input with some values just above and some values just below integers shows some kind of precision problem.)

Although the square root scale works well at achieving symmetry, further analysis might use a generalized linear model with log link as being appropriate for counts or count-like outcomes -- except that, as we started, it seems that there is at best a small difference between these groups on this outcome that doesn't achieve conventional significance.

EDIT Some experiments with generalized linear models showed that it doesn't much matter what link (say identity, square root or log) or family (say Gaussian or gamma or Poisson) you specify. A hypothesis of different means yields P-values around 0.15 in every seemingly plausible case tried. The science may run that there should be a notable difference, but you need a bigger sample to establish it conventionally.

• Thanks for additional insight. Commented Apr 7, 2021 at 15:06
• Been thinking about this. 'Nearly' equal sample mean and SD in each group might indicate exponential data. Sample sizes 66 OSA, 51 Non. Crude quantile 95% parametric bootstrap CI of ratio of exponential means is $(1.26, 2.33),$ which does not include $1.$ Minor medical procedure earlier today with a bit of medication, so not at my best right now. If correct, does that CI suggest a worthwhile path to you? Commented Apr 8, 2021 at 5:59
• Exponential might be a good idea, depending partly on the underlying science, about which I know nothing. As said, I already played briefly with the more general gamma distribution. (Best wishes for full and quick recovery.) Commented Apr 8, 2021 at 9:47
• Hi @NickCox, please see my answer below. These plots are revealing, but they don't tell us much about whether the t test should reject or not. Commented Apr 10, 2021 at 16:22
• @user257566 I agree with that. I am just showing ways to look at the data. I also hold that focus on means being significantly different is a vastly over-rated question. Commented Apr 10, 2021 at 16:28

A number by itself is rather meaningless. Is 1000 a large number? A 1000 kg sandwich is really big, but a 1000 mg one is tiny. When we look at the difference between two means, we want to compare it to something, and usually that something is the standard deviation. The basic formula for combining two population standard deviations is like the Pythagorean formula: the combined SD squared is the sum of the SDs squared (there are more complicated formulas to take into account that we don't actually have the population SD but instead are estimating it from the sample, but the differences between them isn't significant here). This gives us $$\sqrt{1079^2+ 922^2}$$ or about 1419. The difference between the means is 278, and 278 is only about 20% of the combined standard deviation, which is rather small. For a two-tailed test, we'd generally need 200% to consider the difference significant.

For the mean standard deviation, we need to also take into account the sample sizes. As result, the SD for the distribution of sample means is smaller than the population standard deviation, and thus the difference to SD ratio is higher, but not enough to reach significance.

• "usually that something is the standard deviation" It might be useful if this post also explains why the standard deviation is used. If I have a group of sandwiches with a mean of 1000 mg and another group with a mean of 1000 kg, then this is a big difference and it is not clear why I should care about the standard deviation. Commented Apr 7, 2021 at 20:53
• Sandwich standard errors are exactly what are needed here for a heteroscedastic set up. en.wikipedia.org/wiki/… Commented Apr 7, 2021 at 21:37

Most of the answers so far have focused on the distribution of the data itself, but that is not the correct reason that the t test is not rejecting. The $$p$$ value of the test you did will be less than $$0.05$$ whenever the test statistic $$\frac{\bar{X}_1 - \bar{X}_2}{\sqrt{\frac{S_1^2}{n_1} + \frac{S_2^2}{n_2}}}$$ is larger in absolute value than roughly $$2$$. The numerator of this test statistic is the difference in sample means, which you correctly point out is high in your case. However, the denominator which is determined by the standard errors of the sample means (i.e. $$S_1^2/n_1$$ and $$S_2^2/n_2$$) matters too. In your case, the standard errors are high enough that the test statistic is too small to reject.

That's all there is to it. Let's run through a series of other questions and evaluate whether they influence the answer:

1. Does the normality of the data matter? No.
2. Does the skewness of the data matter? No.
3. Does the distribution of the data matter? No, except for the sample mean and standard deviation.
4. Does the data standard deviation give you full information about the relevant variability? No, the standard error is what matters, which depends on the sample size.

All you need to know is the difference in sample means $$\bar{X}_1 - \bar{X}_2$$ and each groups standard error of the mean, $$S_1^2/n_1$$ and $$S_2^2/n_2$$. I can't emphasize this enough. No other term appears in the statistic above (and none other appear in the null distribution).

Almost all of the earlier answers have focused exclusively on the data standard deviations (i.e. $$S_1^2$$ and $$S_2^2$$) instead of the standard errors of the sample means (i.e. $$S_1^2/n_1$$ and $$S_2^2/n_2$$). This is a mistake which I hope hasn't led to too much confusion. They've also focused on inspecting whether the t test is appropriate for your data. This is an interesting question which must be known to determine the true type I error rate of the $$t$$ test on your data, but it isn't the one you asked.

A convenient way to visualize these relevant terms is below. Since only the mean and standard errors matter, we are assuming the assumptions of the $$t$$ test for convenience. Each curve below visualizes how much information we have to determine the true mean. Because they overlap considerably, we don't have enough information to separate the two means and so the $$t$$-test does not reject. Note, it does not matter whether the data distributions overlap or not.

• +1 for placing the accent on the distribution of the estimate rather than the distribution of the data. However, your point that normality, skewness, distribution of the data don't matter is a bit simplistic (and should not be considered general beyond the OP's case). The OP's sample is large, but for small samples sizes the distribution of the estimate has some influences from the distribution of the data. It is only for large sample sizes that the distribution of the estimate becomes close to a normal distribution and that the estimate of a standard error makes sense. Commented Apr 10, 2021 at 20:06
• Hi @SextusEmpiricus . I agree that the distribution of the estimate has some influences from the distribution of the data. However, I disagree that this is simplistic in any way. My point was merely that the $t$ test is based on a set of assumptions. Those assumptions lead to a test statistic and its null distribution. Even if someone's data doesn't satisfy those assumptions, the $t$ test is still the same. Indeed, if someone's data violates the $t$ test assumptions, they should just use a different test--certainly the statistic won't be $t$ any more. Commented Apr 10, 2021 at 20:24
• @SextusEmpiricus Put more simply: do you think the output of 𝑡.𝑡𝑒𝑠𝑡 in R would change if mean and sd of the data stayed the same but the skewness, nonnormality, etc all changed drastically? Certainly not. The OP is wondering why the 𝑡 test didn't reject for his data, not whether the $t$ test has its nominal type I error rate (i.e. whether the data is appropriate for the test) Commented Apr 10, 2021 at 20:39