# Significance of difference in means

In “A Comparative Study of Nursing Home Residents …”, Aigner et al. write (p. 20)

Acute visits were significantly higher for the nurse practiticoner/physician team at 3.0 visits per year (± 2.4) versus 1.2 visits per year (± 1.5) for the physician-only group (P <0.0001)

Furthermore, (p. 19)

The statistical analysis was performed using the chi-squared or Fisher exact test for comparisons of percent and Student t test for comparison of means

I always thought that if confidence intervals overlap by that much, differences can’t be significant. Am I wrong?

Aigner, M. J., Drew, S., & Phipps, J. (2004). A comparative study of nursing home resident outcomes between care provided by nurse practitioners/physicians versus physicians only. Journal of the American Medical Directors Association, 5(1), 16-23.

• Reporting possible negative number of visits per years in the descriptive statistics looks a bit odd to me "1.2 visits per year (± 1.5)". – user10525 Apr 20 '12 at 10:28
• They're probably not confidence intervals for the mean because, as you pointed out, overlap of $(1-\alpha) \cdot 100\%$ confidence intervals is equivalent to rejecting the corresponding two-sample test of means at the $\alpha$ level (assuming the same distribution is used to generate both, e.g. $t$-distribution, which was probably used here). – Macro Apr 20 '12 at 12:16
• The thought is correct, Macro, but the application seems a little bit off: [lack of] overlap of confidence intervals is definitely not equivalent to rejecting the hypothesis of equal means at the $\alpha$ level. It's equivalent to a much stronger test than that. – whuber Apr 20 '12 at 17:30

They're probably not confidence intervals for the mean, but rather standard deviations from the data, just reported weirdly.

This interpretation is supported by the very confused presentation of results in the second quote. Fisher's exact test is a) not the same as a chi-squared test, b) almost certainly inappropriate given most sampling schemes (both margins are seldom fixed). And if they used both we'd want to know why. Neither test compares percentages, although probably results are ultimately reported in percentages. Finally, applying Students t with such low counts seems risky at best.

From the other side of the paywall its hard to say more, but I think you're seeing a weak analysis ambiguously presented.

• Hi, thanks for your answer. It also seems to me that they are reporting something different than SD of the means. However, in defense of Aigner et al., they do not apply Fischers’exact here but Student’s t test. – mzuba Apr 23 '12 at 10:07

The data aren't normal. I presume that the number of acute visits is an integer, i.e. you can't visit a patient 1.5 times. You either visit them once or twice.

As an example, here are some data:

Mean: 3.2 sd: 2.142
8 8 4 1 2 2 0 2 5 2 3 3 3 1 5 4 4 1 4 2


and

Mean:1.25 sd: 1.164
4 0 4 1 1 2 0 1 2 2 1 2 1 0 0 0 1 1 1 1


If you performed a t-test, you would get p=0.001

• At first I thought this reply did not address the question, but then I realized it's pointing out that a t-test is probably inappropriate because the evidence is that the data are (a) few in number and (b) highly skewed. (Normality isn't really an issue with t-tests; normality of the sampling distribution of the mean is what matters.) Interestingly, a permutation test of the mean for these data gives a one-sided p-value of 0.00038 (n=100,000), suggesting a two-sided p-value around 0.0008, which is exactly what a Wilcoxon test produces. The t-test lacks a little power, but seems ok here. – whuber Apr 20 '12 at 17:40
• @whuber How do you perform the permutation test of the mean? – Antoni Parellada Nov 23 '15 at 5:57
• @Antoni Resample repeatedly from the combined dataset. In R this could be done efficiently as x <- scan(text="8 8 4 1 2 2 0 2 5 2 3 3 3 1 5 4 4 1 4 2"); y <- scan(text="4 0 4 1 1 2 0 1 2 2 1 2 1 0 0 0 1 1 1 1"); xy <- c(x, y); x.n <- length(x); s <- replicate(1e5, sum(sample(xy, x.n))); s <- s / x.n - (sum(xy) - s) / length(y) stat <- mean(x) - mean(y); mean(s > stat + 1/2*(s == stat)). It takes less than one second. – whuber Nov 23 '15 at 16:33