# 95% CI for GMC did not overlap but paired Wilcoxon signed rank test gave a p-value > 0.05

I measured antibodies for a pathogen on the same group of subjects measured at month 2 and month 7. Since antibody concentrations vary a lot from subject to subject, we decided to use Geometric Mean Concentration and its 95% CI to summarize the data. Here are the results: at month 2: GMC 34.6 [25.7-46.5] and at month 7 GMC 81.3 [70.7-93.7].

However, when I conducted the paired Wilcoxon signed rank test, the given p-value was 0.17 which shows a non-significant difference. Can anyone help to explain what is going on here?

Here are my data (on original scale):

m2PRNaP = c( 6.413, 830.555, 81.209, 10.772, 53.663, 183.996, 2.5, 23.498, 25.232, 28.811, 229.626, 83.899, 2.5, 18.683, 508.658, 163.899, 15.064, 60.402, 2.5, 5.783, 8.158, 11.747, 23.672, 6.33, 102.674, 38.869, 85.768, 5.953, 44.303, NA, 228.719, 2.5, 67.203, 2.5, 1035.267, 56.614, 215.367, 554.776, 43.772, 2.5, 938.538, 2.5, 2.5, 2.5, 2.5, 2.5, 2.5, 197.033, 33.46, 2.5, 162.874, 2.5,32.354, 9.242, 60.039, 121.752, 45.975, 77.019, 86.576, 81.666, 5.642, 79.18, 19.839,24.658, 429.635, 13.589, 2.5, 509.928, 31.169, 192.494, 5.474, 101.242, 1114.844, 705.47, 16.918, 321.437, 2.5, 5.322, 562.846, 62.267, 63.529, 2.5, 7.582, 2.5, 2.5, 29.382, 142.736, 2.5, 82.015, 1201.641, 416.918, 15.617, 6.665,10.544, 73.719, 101.031, 2.5, 44.038, 12.457, 167.542, 215.367, 6.535, 30.943, 963.577, 388.875, 43.661, 34.73, 62.958, 104.67, 49.644, NA, 14.976,2.5,8.902, 76.642, 92.634, 390.613, 2.5, 21.804, 19.073, 14.901, 17.681, 360.894, 16.811, 15.41, 163.207, 37.933, 280.794, 427.119, 260.464, 285.483, 99.846, 5.082, 508.658, 416.918, 119.076, 5.783, 2.5, 2.5, 21.746, 13.524, 22.251, 62.094, 2.5, 165.398, 437.104)

m7PRNaP = c( 61.347, 92.082, 13.383, 406.484, NA, 212.318, NA, NA, 27.857, NA, 74.79, 17.51, 807.83, 191.266, 393.082, 27.68, 39.419, 202.477, 15.224, 47.647, 60.228, 136.9, 96.243, 55.122, 77.531, 61.582, 103.415, 184.796, 267.117, 165.398, 109.04, 77.997, 82.054, 250.578, 100.383, 44.752, NA, NA, 27.174, 115.069, 62.421, 85.095, 110.458, 163.153, 388.135, 25.964,146.523,NA,37.322, 114.46, 40.439, 150.763, 23.359, 79.744, 70.82, 37.882, 91.598, 43.936, 30.613, 468.872, 31.386, 53.658, 410.917, 11.68, 51.463, 249.148, 177.279, 70.15, 23.646, 17.046, 160.556, 38.766, 48.334, 94.554, 206.722, NA, 76.277, 244.048, 135.891, 76.917, 74.289, 105.249, 64.361, 94.62, 93.154, 106.413, 214.057, 68.019, 88.545, 65.594, 363.356, 60.84, 85.855, 39.503, 31.163, 72.406, 135.111, 248.673, 121.266, 71.494, 47.573, 28.529, 26.413, 419.991, NA, 77.244, 202.055, 64.334, 134.218, 69.993, 68.004, 151.119, 277.136, 48.459, 37.012, 42.123, 73.734, 50.957, 70.988, 229.172, NA, 47.21, 99.658, 74.4, 37.54, 21.612, 62.288, 35.441, 135.891, 89.095, 24.001, 64.113, NA, 270.099, 22.038, 18.673, 88.235, 166.15, 249.148, 57.545, 197.865, 82.884, 74.801, 186.402, 39.885, 136.226)


Here are the codes I used to calculate the GMC and its 95% CI based on some discussion here: How to calculate confidence interval for a geometric mean?

gmc2 <- function(anti_logscale){ # vector of the antibody on logscale
mean_gmc <- exp(mean(anti_logscale))
n  <- length(anti_logscale)
s2y<- sd(anti_logscale)/sqrt(length(anti_logscale))
ciup <- exp(mean(anti_logscale) + 1.96*s2y)
cilow<- exp(mean(anti_logscale) - 1.96*s2y)
return(list(mean_gmc, cilow,ciup,n))
}
gmc2(log(m2PRNaP[complete.cases(m2PRNaP)])) # 34.6 [25.7-46.5]
gmc2(log(m7PRNaP[complete.cases(m7PRNaP)])) # 81.3 [70.7-93.7]


Now if I use the paired Wilcoxon signed rank test to test the difference between antibody levels at month 2 and month 7:

wilcox.test(m2PRNaP,m7PRNaP, paired= TRUE) # p-value of 0.17


Any contribution would be great. I was thinking maybe the log-transformed data at month 2 did not follow a normal distribution might lead to the problem. But since the sample size of 144, we can rely on the Central Limit Theorem, right?

P/s: Here are the histograms of the data at two time points. Boxplots (on original scale and log-scale for both are also added).

• I think the answer to this is quite simple: the Wilcoxon test is not a test of geometric mean difference. – AdamO Jan 3 '18 at 14:46
• @AdamO: I agree on this point. The Wilcoxon signed rank test was used to compare the antibody levels on their original scales. The test of geometric mean difference was not done since the data seemed not to be log-normally distributed (I was looking at the data on log scale using a qqplot and the plots did not suggest that the log-transformed data follow a normal distribution). Any idea to still proceed a test of geometric mean difference in this case? – bienco88 Jan 4 '18 at 10:49
• Do the test anyway. The CLT still applies to log transformed data. You have a huge sample to fear that finite sample properties are of any concern. The log transform is very intuitive for measuring associations or differences in concentration data. – AdamO Jan 4 '18 at 14:56
• Indeed, I did the paired Wilcoxon signed rank test on the log-transformed data and the p-value was < 0.0001 in the example I gave. – bienco88 Jan 4 '18 at 15:14
• Ah I didn't make myself clear: Do the simple T test of the log transformed data because it is a test of geometric mean difference. If the T test here is not significant, then you have incorrectly calculated the CIs that you reported above. They should agree 1-1. The Wilcoxon does not agree for the reasons I mentioned earlier. – AdamO Jan 4 '18 at 15:21

## 2 Answers

You have several problems here.

1 - you are computing confidence intervals for each time point separately but you are testing the differences which is not the same thing.

2 - you are removing missing values from your computation in different ways for the confidence intervals and for the test.

3 - you are not in fact computing the Mann Whitney U which is for independent samples but rather the Wilcoxon signed ranks test. This is not affecting your results: you are actually doing the right thing but not naming it correctly.

Are you sure the values you present are on the log scale? When exponentiated they become very large. Even if they are base 2 logs they are still pretty big.

• Thank you. 1. I understand the things that you are mentionning. However, as far as I know, if you are computing CI for each time point separately, and they are overlapped, there can be either significant or not significant difference using formal testing. However, if the two CIs don't overlap, it is sure that the test of difference should lead to a p-value < 0.05. Do you agree? 2. In both calculations, missing values were excluded. 3. I will edit my question. Data are on original scale. – bienco88 Jan 3 '18 at 12:31
• As far as your points in the comment go: point 1 is false you are confusing independent and dependent samples, 2 you have not excluded them in the same way, you did it once per variable and once listwise. – mdewey Jan 3 '18 at 14:45

If there's a pattern here it's that people low at month 2 are usually higher later and those high at month 7 are lower later. Any situation in which some are better and some worse systematically is not well matched by just looking for a shift in geometric mean, or indeed a shift in any other single summary. But there are more of the former than the latter, so the geometric mean does increase.

I can't see why Mann-Whitney is thought to be of any use or interest. The data look fairly well behaved on logarithmic scale, and so geometric mean is a naturally relevant summary; looking at ranks alone just throws away detail that is likely to be important medically (biologically) as well as statistically.

That said, the spike of values at 2.5 in month 2 (I can't give the units of measurement because you don't cite them) requires a little comment. If it's a reflection of measurement problems, many of the other values may be spuriously precise. If it's imputed somehow, those cases are possibly useless for comparison. The dropout from month 2 to month 7 may also deserve comment: at the starkest, perhaps some of the patients died.

I am totally lay on the medical questions here.

A scatter plot using logarithmic scales is an obvious plot to consider. The diagonal line is a line of equality separating cases of increase from those of decrease.

Beyond that a plot the ratio (month 7/month 2) on a logarithmic scale (equivalent to a difference of logarithms) versus the original seems to help. The reference line here is that of ratio 1.

Disclaimer: I don't use R routinely and didn't try to follow exactly what you did. It's not important, but the graphs were drawn in Stata.

• Never compute individual confidence intervals. It doesn't matter if these overlap or not. Directly compute the confidence interval for the difference. This can exclude zero even with the two irrelevant confidence intervals overlap. – Frank Harrell Jan 3 '18 at 12:07
• Thank you @Nick Cox for your thorough answer. Yes, there are some values that fell below the limit of detection (5 IU/mL) and it is decided to impute the LOD/2 = 2.5 for those values. That is why you could see many measurements of 2.5 – bienco88 Jan 3 '18 at 12:10
• @FrankHarrell Naturally I agree with you, but I think your comment is on the original question, not on anything here. – Nick Cox Jan 3 '18 at 12:13
• @Nick Cox: We used the GMC and 95% CI to describe the data and compare antibody levels at different time points. However, our clinicians wanted to have a formal test with p-values. Since our data did not follow a normal distribution, we decided to go with a non-parametric testing, hence (paired) Mann Whitney U test was used. – bienco88 Jan 3 '18 at 12:21
• Because there is a systematic pattern to the change. You can identify empirical thresholds with three zones either all go up or all go down in the outer zones, Mushing that together misses the main message of your data. As @Frank Harrell pointed out w.r.t. confidence intervals, looking just at marginal distributions for 2 months and 7 months ignores a key part of the information. The pairing of points for each patient is absolutely vital, but it's ignored by your plots. – Nick Cox Jan 3 '18 at 13:51