Why the p-value of t.test() is not statistically significant when mean values look really different

Question

I am trying to find out whether there is a significant difference in the mean value of a biomarker between two groups. I am using t.test in R. The mean(SD) values are 1142(1079) and 864(922)in the groups. But the p-values of the test shows the difference is not statistically significant. Can someone please help me? I am sharing the dput of the dataframe below.

structure(list(ANGPTL7 = c(2.5, 205, 885, 1915, 835, 1685, 625, 
1615, 84.9999999999999, 1175, 2695, 235, 1025, 2.5, 2915, 825, 
255, 1085, 1815, 2.5, 205, 985, 2.5, 705, 435, 555, 2045, 135, 
15, 975, 2285, 1905, 515, 74.9999999999999, 25, 815, 1075, 2.5, 
1115, 3115, 64.9999999999999, 64.9999999999999, 325, 595, 285, 
2.5, 2.5, 345, 5.00000000000001, 215, 3465, 555, 855, 3745, 25, 
305, 2.5, 2.5, 15, 115, 565, 94.9999999999999, 1005, 575, 405, 
2.5, 1855, 1795, 145, 2555, 1705, 74.9999999999999, 735, 375, 
2.5, 475, 1675, 1105, 345, 385, 3195, 115, 1475, 205, 545, 1265, 
485, 1135, 2595, 3305, 305, 575, 1415, 2925, 3125, 2795, 3125, 
1775, 1125, 15, 1695, 1225, 1625, 3175, 3185, 1445, 3065, 785, 
855, 1115, 145, 595, 435, 185, 345, 2455, 1885), OSA_status = c("Non-OSA", 
"OSA", "Non-OSA", "OSA", "Non-OSA", "OSA", "Non-OSA", "OSA", 
"Non-OSA", "Non-OSA", "Non-OSA", "OSA", "OSA", "OSA", "OSA", 
"OSA", "Non-OSA", "Non-OSA", "OSA", "OSA", "Non-OSA", "Non-OSA", 
"OSA", "Non-OSA", "OSA", "OSA", "OSA", "OSA", "Non-OSA", "Non-OSA", 
"Non-OSA", "OSA", "OSA", "OSA", "OSA", "Non-OSA", "OSA", "OSA", 
"OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA", 
"OSA", "Non-OSA", "Non-OSA", "Non-OSA", "OSA", "Non-OSA", "OSA", 
"OSA", "OSA", "OSA", "Non-OSA", "Non-OSA", "OSA", "OSA", "Non-OSA", 
"OSA", "OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA", 
"OSA", "OSA", "OSA", "OSA", "OSA", "OSA", "OSA", "OSA", "OSA", 
"OSA", "OSA", "Non-OSA", "OSA", "Non-OSA", "OSA", "Non-OSA", 
"Non-OSA", "OSA", "Non-OSA", "OSA", "Non-OSA", "OSA", "OSA", 
"OSA", "OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA", "Non-OSA", 
"OSA", "Non-OSA", "Non-OSA", "Non-OSA", "OSA", "Non-OSA", "OSA", 
"OSA", "OSA", "OSA", "Non-OSA", "Non-OSA", "OSA", "OSA", "OSA", 
"OSA", "OSA", "OSA", "Non-OSA", "OSA", "OSA")), row.names = c(NA, 
-117L), class = c("tbl_df", "tbl", "data.frame"))

EDIT This data example is great for R users but a small or large pain for everyone else. This format may or may not be easier, depending.

id  ANGPTL7 OSA_status
1       2.5    Non-OSA
2     205.0        OSA
3     885.0    Non-OSA
4    1915.0        OSA
5     835.0    Non-OSA
6    1685.0        OSA
7     625.0    Non-OSA
8    1615.0        OSA
9      85.0    Non-OSA
10   1175.0    Non-OSA
11   2695.0    Non-OSA
12    235.0        OSA
13   1025.0        OSA
14      2.5        OSA
15   2915.0        OSA
16    825.0        OSA
17    255.0    Non-OSA
18   1085.0    Non-OSA
19   1815.0        OSA
20      2.5        OSA
21    205.0    Non-OSA
22    985.0    Non-OSA
23      2.5        OSA
24    705.0    Non-OSA
25    435.0        OSA
26    555.0        OSA
27   2045.0        OSA
28    135.0        OSA
29     15.0    Non-OSA
30    975.0    Non-OSA
31   2285.0    Non-OSA
32   1905.0        OSA
33    515.0        OSA
34     75.0        OSA
35     25.0        OSA
36    815.0    Non-OSA
37   1075.0        OSA
38      2.5        OSA
39   1115.0        OSA
40   3115.0    Non-OSA
41     65.0    Non-OSA
42     65.0    Non-OSA
43    325.0    Non-OSA
44    595.0    Non-OSA
45    285.0        OSA
46      2.5    Non-OSA
47      2.5    Non-OSA
48    345.0    Non-OSA
49      5.0        OSA
50    215.0    Non-OSA
51   3465.0        OSA
52    555.0        OSA
53    855.0        OSA
54   3745.0        OSA
55     25.0    Non-OSA
56    305.0    Non-OSA
57      2.5        OSA
58      2.5        OSA
59     15.0    Non-OSA
60    115.0        OSA
61    565.0        OSA
62     95.0    Non-OSA
63   1005.0    Non-OSA
64    575.0    Non-OSA
65    405.0    Non-OSA
66      2.5    Non-OSA
67   1855.0        OSA
68   1795.0        OSA
69    145.0        OSA
70   2555.0        OSA
71   1705.0        OSA
72     75.0        OSA
73    735.0        OSA
74    375.0        OSA
75      2.5        OSA
76    475.0        OSA
77   1675.0        OSA
78   1105.0    Non-OSA
79    345.0        OSA
80    385.0    Non-OSA
81   3195.0        OSA
82    115.0    Non-OSA
83   1475.0    Non-OSA
84    205.0        OSA
85    545.0    Non-OSA
86   1265.0        OSA
87    485.0    Non-OSA
88   1135.0        OSA
89   2595.0        OSA
90   3305.0        OSA
91    305.0        OSA
92    575.0    Non-OSA
93   1415.0    Non-OSA
94   2925.0    Non-OSA
95   3125.0    Non-OSA
96   2795.0    Non-OSA
97   3125.0        OSA
98   1775.0    Non-OSA
99   1125.0    Non-OSA
100    15.0    Non-OSA
101  1695.0        OSA
102  1225.0    Non-OSA
103  1625.0        OSA
104  3175.0        OSA
105  3185.0        OSA
106  1445.0        OSA
107  3065.0    Non-OSA
108   785.0    Non-OSA
109   855.0        OSA
110  1115.0        OSA
111   145.0        OSA
112   595.0        OSA
113   435.0        OSA
114   185.0        OSA
115   345.0    Non-OSA
116  2455.0        OSA
117  1885.0        OSA

Answer 1

I agree with @pikachu that the standard deviations are too large compared with the difference between means for a t test to find a significant difference.

Thank you for posting your data. It is always a good idea to take a look at some graphic displays of the data before doing formal tests.

Stripcharts of observations in the two groups do not show a meaningful difference in locations relative to the variability of the samples.

stripchart(ANGPTL7 ~OSA_status, pch="|", ylim=c(.5,2.5))

Here are boxplots of the two groups. The 'notches' in the the sides of the boxes are nonparametric confidence intervals, calibrated so that overlapping notches tend to indicate no significant difference in location.

boxplot(ANGPTL7 ~ OSA_status, notch=T, 
        col="skyblue2", horizontal=T)

Even with sample sizes as large as these, I would be reluctant to do a two-sample t test on account of the marked skewness of the data. I would do a nonparametric two-sample Wilcoxon rank sum test (which also shows no significant difference).

wilcox.test(ANGPTL7 ~ OSA_status)

        Wilcoxon rank sum test with continuity correction

data:  ANGPTL7 by OSA_status
W = 1456.5, p-value = 0.2139
alternative hypothesis: true location shift is not equal to 0

Answer 2

When you consider the difference between means you have to use a different unit than the simple absolute difference. Take into account that you are measuring the difference in means produced by two random sources. Theose random sources (whose outcomes are your two samples) contains variability. It is this variability which should be used to compare the difference in means. The standard deviations are measured in the same units as the difference so you should see the difference in terms of standard deviations.

Now you see that the difference in means $|1142-864|$ is much lower than any of the sample standard deviations by far. If you approximate with a normal distribution than you will see that the difference in means is about one third of a standard deviation (whatever one can be used) and you can see that there are a lot of chances that the difference in mean to be produced simply by random errors. Use as an approximation that in normal distribution with one standard deviation on the left of mean and on the right you will get aproximately 60% of your data, and with 2 standard deviations you will cover around 95% of your data. The mean difference is far less than any standard deviations so you should consider them equal.

To illustrate the idea of using the mean difference and variation for comparison you can take a look at this image When you compare the means taking into account the variance you see that the two distributions are not very different, their overlap is large.

When you work out the t test for difference the distribution for the difference in means looks like the following: The vertical bar is where your mean difference is in terms of standard deviations and the distribution displays how it will variate. It should be clar that the value of the standard deviation scaled difference in mean is not an exceptional value and as a consequence it cannot produce a small p-value.

Answer 3

With respect to plotting the data, I'd like to point the R package ggbeeswarm. In this case I think it is better than a boxplot or violin plot. The horizontal segments are the quantiles at 5%, 50%, 95%; dat is the data from the OP

library(ggplot2)
library(ggbeeswarm)
library(data.table)

dat <- as.data.table(dat)
dat[, OSA_status := as.factor(OSA_status)]
qq <- dat[, list(quantile= quantile(ANGPTL7, p= c(0.05, 0.5, 0.95))), by= OSA_status]

gg <- ggplot(data= dat, aes(x= OSA_status, y= ANGPTL7)) +
    geom_quasirandom(width= 0.25) +
    geom_segment(data= qq, aes(y= quantile, yend= quantile, x= as.numeric(OSA_status)-0.1, xend= as.numeric(OSA_status)+0.1), colour= 'blue') +
    theme_light()

Answer 4

You have too high standard deviation (uncertainty) and with selected probability (I assume $\alpha = 0.05$ that is by default) the confidence intervals are overlapping, thus no statistically significant difference between means.

EDIT: CI of one mean covers the estimate of the second mean, thus they are not significantly different. Thanks for correction from comments.

Answer 5

Excellent points in other answers. I went a little further in looking at the data. The common practice of firing up box plots when the question is about means is a good idea because a good box plot will give you a good overview of the data, but it is a little indirect whenever -- as is a typical default in software -- the means aren't shown at all, but have to be guessed at, or inserted mentally from another calculation, or indeed added whenever the software allows. I started out with something a little like @BruceET's display, considered a logarithmic transformation and then retreated because that went too far. A square root scale seemed to work better for visualization.

This display is a hybrid quantile and box plot using square root scale. Each box plot is minimal and shows only minimum, maximum, median and quartiles, for all of which it is true in principle that summary(root()) $=$ root(summary()). The box plot is minimal because alongside each a quantile plot is shown with all the observed values in order. The horizontal reference line is at the mean of the square roots, which is close to the median in each case.

Fine structure is revealed by the quantile plot that the box plot conceals, and an easy tabulation shows that all values are multiples of 5, except that there is a minor mode at the lowest reported value, 2.5. That leads me to guess at a detection limit problem, such that any observed zero has been fudged, nudged or kludged to half the smallest non-zero value of 5. (I take it that the data input with some values just above and some values just below integers shows some kind of precision problem.)

Although the square root scale works well at achieving symmetry, further analysis might use a generalized linear model with log link as being appropriate for counts or count-like outcomes -- except that, as we started, it seems that there is at best a small difference between these groups on this outcome that doesn't achieve conventional significance.

EDIT Some experiments with generalized linear models showed that it doesn't much matter what link (say identity, square root or log) or family (say Gaussian or gamma or Poisson) you specify. A hypothesis of different means yields P-values around 0.15 in every seemingly plausible case tried. The science may run that there should be a notable difference, but you need a bigger sample to establish it conventionally.

Answer 6

A number by itself is rather meaningless. Is 1000 a large number? A 1000 kg sandwich is really big, but a 1000 mg one is tiny. When we look at the difference between two means, we want to compare it to something, and usually that something is the standard deviation. The basic formula for combining two population standard deviations is like the Pythagorean formula: the combined SD squared is the sum of the SDs squared (there are more complicated formulas to take into account that we don't actually have the population SD but instead are estimating it from the sample, but the differences between them isn't significant here). This gives us $\sqrt{1079^2+ 922^2}$ or about 1419. The difference between the means is 278, and 278 is only about 20% of the combined standard deviation, which is rather small. For a two-tailed test, we'd generally need 200% to consider the difference significant.

For the mean standard deviation, we need to also take into account the sample sizes. As result, the SD for the distribution of sample means is smaller than the population standard deviation, and thus the difference to SD ratio is higher, but not enough to reach significance.

Answer 7

Most of the answers so far have focused on the distribution of the data itself, but that is not the correct reason that the t test is not rejecting. The $p$ value of the test you did will be less than $0.05$ whenever the test statistic $$\frac{\bar{X}_1 - \bar{X}_2}{\sqrt{\frac{S_1^2}{n_1} + \frac{S_2^2}{n_2}}}$$ is larger in absolute value than roughly $2$. The numerator of this test statistic is the difference in sample means, which you correctly point out is high in your case. However, the denominator which is determined by the standard errors of the sample means (i.e. $S_1^2/n_1$ and $S_2^2/n_2$) matters too. In your case, the standard errors are high enough that the test statistic is too small to reject.

That's all there is to it. Let's run through a series of other questions and evaluate whether they influence the answer:

1. Does the normality of the data matter? No.
2. Does the skewness of the data matter? No.
3. Does the distribution of the data matter? No, except for the sample mean and standard deviation.
4. Does the data standard deviation give you full information about the relevant variability? No, the standard error is what matters, which depends on the sample size.

All you need to know is the difference in sample means $\bar{X}_1 - \bar{X}_2$ and each groups standard error of the mean, $S_1^2/n_1$ and $S_2^2/n_2$. I can't emphasize this enough. No other term appears in the statistic above (and none other appear in the null distribution).

Almost all of the earlier answers have focused exclusively on the data standard deviations (i.e. $S_1^2$ and $S_2^2$) instead of the standard errors of the sample means (i.e. $S_1^2/n_1$ and $S_2^2/n_2$). This is a mistake which I hope hasn't led to too much confusion. They've also focused on inspecting whether the t test is appropriate for your data. This is an interesting question which must be known to determine the true type I error rate of the $t$ test on your data, but it isn't the one you asked.

A convenient way to visualize these relevant terms is below. Since only the mean and standard errors matter, we are assuming the assumptions of the $t$ test for convenience. Each curve below visualizes how much information we have to determine the true mean. Because they overlap considerably, we don't have enough information to separate the two means and so the $t$-test does not reject. Note, it does not matter whether the data distributions overlap or not.