10
$\begingroup$

I have data as two lists:

acol = [8.48, 9.82, 9.66, 9.81, 9.23, 10.35, 10.08, 11.05, 8.63, 9.52, 10.88, 10.05, 10.45, 10.0, 9.97, 12.02, 11.48, 9.53, 9.98, 10.69, 10.29, 9.74, 8.92, 11.94, 9.04, 11.42, 8.88, 10.62, 9.38, 12.56, 10.53, 9.4, 11.53, 8.23, 12.09, 9.37, 11.17, 11.33, 10.49, 8.32, 11.29, 10.31, 9.94, 10.27, 9.98, 10.05, 10.07, 10.03, 9.12, 11.56, 10.88, 10.3, 11.32, 8.09, 9.34, 10.46, 9.35, 11.82, 10.29, 9.81, 7.92, 7.84, 12.22, 10.42, 10.45, 9.33, 8.24, 8.69, 10.31, 11.29, 9.31, 9.93, 8.21, 10.32, 9.72, 8.95, 9.49, 8.11, 8.33, 10.41, 8.38, 10.31, 10.33, 8.83, 7.84, 8.11, 11.11, 9.41, 9.32, 9.42, 10.57, 9.74, 11.35, 9.44, 10.53, 10.08, 10.92, 9.72, 7.83, 11.09, 8.95, 10.69, 11.85, 10.19, 8.49, 9.93, 10.39, 11.08, 11.27, 8.71, 9.62, 11.75, 8.45, 8.09, 11.54, 9.0, 9.61, 10.82, 10.36, 9.22, 9.36, 10.38, 9.53, 9.2, 10.36, 9.38, 7.68, 9.99, 10.61, 8.81, 10.09, 10.24, 9.21, 10.17, 10.32, 10.41, 8.77]

bcol = [12.48, 9.76, 9.63, 10.86, 11.63, 9.07, 12.01, 9.52, 10.05, 8.66, 10.85, 9.87, 11.14, 10.59, 9.24, 9.85, 9.62, 11.54, 11.1, 9.38, 9.24, 9.68, 10.02, 9.91, 10.66, 9.7, 11.06, 9.27, 9.08, 11.31, 10.9, 10.63, 8.98, 9.81, 9.69, 10.71, 10.43, 10.89, 8.96, 9.74, 8.33, 11.45, 9.61, 9.59, 11.25, 9.44, 10.05, 11.63, 10.16, 11.71, 9.1, 9.53, 9.76, 9.33, 11.53, 11.59, 10.21, 10.68, 8.99, 9.44, 9.82, 10.35, 11.22, 9.05, 9.18, 9.57, 11.43, 9.4, 11.45, 8.39, 11.32, 11.16, 12.47, 11.62, 8.77, 11.34, 11.77, 9.53, 10.54, 8.73, 9.97, 9.98, 10.8, 9.6, 9.6, 9.96, 12.17, 10.01, 8.69, 8.94, 9.24, 9.84, 10.39, 10.65, 9.31, 9.93, 10.41, 8.5, 8.64, 10.23, 9.94, 10.47, 8.95, 10.8, 9.84, 10.26, 11.0, 11.22, 10.72, 9.14, 10.06, 11.52, 10.21, 9.82, 10.81, 10.3, 9.81, 11.48, 8.51, 9.55, 10.41, 12.17, 9.9, 9.07, 10.51, 10.26, 10.62, 10.84, 9.67, 9.75, 8.84, 9.85, 10.41, 9.18, 10.93, 11.41, 9.52]

A summary of the above lists is given below:

N,   Mean, SD,   SEM,   95% CIs
137  9.92  1.08  0.092  (9.74, 10.1)
137  10.2  0.951 0.081  (10.0, 10.3)

An unpaired t-test for the above data gives a p-value of 0.05:

f,p = scipy.stats.ttest_ind(acol, bcol)
print(f, p)
-1.9644209241736 0.050499295018989004

I understand from this and other pages that mean ± 2 * SEM (standard error of mean as calculated by SD/sqrt(N)) gives a 95% confidence interval (CI) range.

I also believe that if 95% confidence intervals are overlapping, P-value will be > 0.05.

I plotted the above data as mean ± 2 * SEM:

Enter image description here

The 95% confidence intervals are overlapping. So why is the p-value reaching a significant level?

$\endgroup$
  • 5
    $\begingroup$ See stats.stackexchange.com/a/18259/919 for a detailed analysis of this situation. It shows your belief is incorrect but it can be patched up by adjusting the p-value. $\endgroup$ – whuber Nov 21 '20 at 14:47
  • 4
    $\begingroup$ +1 Good question because it shows good understanding of the relationship between hypothesis testing and confidence interval $\endgroup$ – Patrick Coulombe Nov 21 '20 at 21:47
10
$\begingroup$

The overlap is just a (strict/inaccurate) rule of thumb

The point when the error bars do not overlap is when the distance between the two points is equal to $2(SE_1+SE_2)$. So effectively you are testing whether some sort of standardized score (distance divided by the sum of standard errors) is greater than 2. Let's call this $z_{overlap}$

$$ z_{overlap} = \frac{\vert \bar{X}_1- \bar{X}_2 \vert}{SE_1+SE_2} \geq 2$$

If this $z_{overlap} \geq 2$ then the error bars do not overlap.

enter image description here


The standard deviation of a linear sum of independent variables

Adding the standard deviations (errors) together is not the typical way to compute the standard deviation (error) of a linear sum (the parameter $\bar{X}_1-\bar{X}_2$ can be considered as a linear sum where one of the two is multiplied by a factor $-1$) See also: Sum of uncorrelated variables

So the following are true for independent $\bar{X}_1$ and $\bar{X}_2$:

$$\begin{array}{} \text{Var}(\bar{X}_1-\bar{X}_2) &=& \text{Var}(\bar{X}_1) + \text{Var}(\bar{X}_2)\\ \sigma_{\bar{X}_1-\bar{X}_2}^2 &=& \sigma_{\bar{X}_1}^2+\sigma_{\bar{X}_2}^2\\ \sigma_{\bar{X}_1-\bar{X}_2} &=& \sqrt{\sigma_{\bar{X}_1}^2+\sigma_{\bar{X}_2}^2}\\ \text{S.E.}(\bar{X}_1-\bar{X}_2) &=& \sqrt{\text{S.E.}(\bar{X}_1)^2 + \text{S.E.}(\bar{X}_2)^2}\\ \end{array}$$

But not

$$\text{S.E.}(\bar{X}_1-\bar{X}_2) \neq {\text{S.E.}(\bar{X}_1) + \text{S.E.}(\bar{X}_2)}$$


'Correct' formula for comparing the difference in the mean of two samples

For a t-test to compare the difference in means of two populations, you should be using a formula like

  • In the simplest case: $$t = \frac{\bar{X}_1 - \bar{X}_2}{\sqrt{SE_1^2+SE_2^2}}$$ this is when we consider the variances to be unequal or when the sample sizes are equal.

  • If the sample sizes are different and you consider the variance of the populations to be equal, then you can estimate the variances for both samples together instead of separately, and use one of many formulae for the pooled variance like

    $$s_p = \sqrt{\frac{(n_1-1)s_1^2 +(n_2-1)s_2^2}{n_1+n_2-2}}$$

    with $$t = \frac{\bar{X}_1 - \bar{X}_2}{s_p \sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}$$

    and with $SE_1 = s_1/\sqrt{n_1}$ and $SE_2 = s_2/\sqrt{n_2}$ you get

    $$t = \frac{\bar{X}_1 - \bar{X}_2}{\sqrt{\frac{n_1+n_2}{n_1+n_2-2} \left( \frac{n_1-1}{n_2} SE_1^2 + \frac{n_2-1}{n_1} SE_2^2 \right)}}$$

Note that the value $\sqrt{SE_1^2+SE_2^2}$ is smaller than $SE_1+SE_2$, therefore $t>z_{overlap}$.

Sidenotes:

  • In the case of the pooled variance, you might have a situation - although it is rare - that the variance of the larger sample is larger than the variance of the smaller sample, and then it is possible that $t<z_{overlap}$.

  • Instead of z-values and a z-test you are actually doing (should be doing) a t-test. So it might be that the levels on which you base the confidence intervals for the error bars (like '95% is equivalent to 2 times the standard error') will be different for the t-test. To be fair, to compare apples with apples, you should use the same standard and base the confidence levels for the error bars on a t-test as well. So let's assume that also for the t-test the boundary level that relates to 95% is equal to or less than 2 (this is the case for sample sizes larger than 60).

If this $t \geq 2$ then the difference is significant (at a 5% level).

The standard error of the difference between two variables is not the sum of standard errors of each variable. This sum is overestimating the error for the difference and will be too conservative (too often claim there is no significant difference).

So $t>z_{overlap}$ and may lead to a significant difference while the error bars have overlap. You do not need non-overlapping error bars in order to have a significant difference. This overlap is a stricter requirement and happens when the p-value is $\leq 0.05$ (and it will often be a lower p-value).

$\endgroup$
  • $\begingroup$ It will be great if you could explain your long sentence / short para on s1+s2, its double, distance and 2*SEM. That will help us understand this concept better. $\endgroup$ – rnso Nov 21 '20 at 17:25
  • $\begingroup$ I expected standard deviation (even pooled) to be much greater than (SEM1+SEM2). I tried with random data and find that it is always larger. Please confirm that you really mean that 2*pooledSD may be less than 2*(SEM1+SEM2) ? $\endgroup$ – rnso Nov 22 '20 at 12:14
  • $\begingroup$ @rnso yes that is really what I mean. You can see this also in the inequality. If you are comparing the difference of two variables $X_1-X_2$, with standard errors $\sigma_1$ and $\sigma_2$, then the standard error of this difference will be smaller than $\sigma_1+\sigma_2$. $\endgroup$ – Sextus Empiricus Nov 22 '20 at 15:00
  • $\begingroup$ How can we calculate standard error of difference if X1 and X2 are unpaired? Sp/sqrt(n1+n2)? $\endgroup$ – rnso Nov 22 '20 at 16:29
  • $\begingroup$ @rnso For any two independent variables $X_1$ and $X_2$ with sample variances $\sigma_1^2$ and $\sigma_2^2$ the sample variance of the difference $X_1-X_2$ is equal to $\sigma_1^2+\sigma_2^2$. See en.wikipedia.org/wiki/… The size of the samples does not matter. $\endgroup$ – Sextus Empiricus Nov 22 '20 at 17:17
3
$\begingroup$

The p-value should be considered between a CI and a parameter value, not two CIs. Indeed, the red point falls entirely outside the blue CI, and the blue point falls entirely outside the red CI.

And it is true that under the null hypothesis such an event would happen 5% of the time:

  • 2.5% of the time, you get a point above the 95% CI
  • 2.5% of the time, you get a point below the 95% CI

If it is only the whiskers that overlap or touch, then the null hypothesis will produce this result a lot less often than 5%. This is because (to use your example) both the blue sample would need to be low, and at the same time the red sample would need to be high (exactly how high would depend on the blue value). You can picture it as a 3D multivariate Gaussian plot, with no skew since the two errors are independent of one another:

enter image description here

Along each axis the probability of falling outside the highlighted region (the CI) is 0.05. But the total probabilities of the blue and pink areas, which gives you P of the two CIs barely touching, is less than 0.05 in your case.

A change of variables from the blue/red axes to the green one will let you integrate this volume using a univariate rather than multivariate Gaussian, and the new variance is the pooled variance from @Sextus-Empiricus's answer.

$\endgroup$
  • 4
    $\begingroup$ I like the image. Unfortunately, you are incorrect at the outset: the relevant comparison is between the CI and the parameter it is intended to estimate, not between the CI and the mean of other data. (That would call for a prediction interval rather than a confidence interval.) $\endgroup$ – whuber Nov 21 '20 at 20:34
2
$\begingroup$

Even if we ignore the difference between confidence and probability, the overlap consists of points for which both the red probability and the blue probability are greater than 0.05. But that doesn't mean that the probability of both is greater than 0.05. For instance, if both the red and blue probability are 0.10, then the joint probability (assuming independence) is 0.01. If you integrate over the whole overlap, this will be less than 0.01.

When you look at the overlap, you are seeing points for which the difference is less than two standard deviations. But remember that the variance of the difference between two variables is the sum of the individual variances. So you can generally use a rule of thumb that if you want to compare two different populations by checking for overlapping CI, you need to divide the size of each CI by $\sqrt 2$: if the variances are of similar sizes, then the variance of the difference will be twice the individual variances, and the standard deviation will be $\sqrt 2$ times as large.

$\endgroup$
  • $\begingroup$ Since each CI is SEM*2, CI/√2 will be SEM*√2. Is there any name given to SEM*√2 or Mean±SEM*√2 ? $\endgroup$ – rnso Nov 22 '20 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.