# Test for mean/median paired difference for differences in proportions (or percentages)

Here is an example of a problem I am trying to figure out: I have data on 20 clinical locations where we recorded how many individuals came for a visit and how many of them chose to get the flu vaccine. Basically, for each of 20 locations, I have proportions. Then, following a flu vaccine awareness campaign, I have data for the same clinical locations again the following year. The total number of patients differs for each clinic between the two years and different clinics are also of different sizes (in terms of patients they saw during that window). So, data looks like:

Clinic 1 (pre/post) 1000/2000 and 1200/1900.
Clinic 2 (pre/post) 450/800 and 500/820.
... Clinic 20 (pre/post)...

Here are some example data in R:

dat <- structure(c(10, 20, 22, 24, 28, 30, 32, 33, 35, 36, 37, 39, 40,
41, 44, 45, 47, 48, 50, 51, 10, 20, 22, 24, 28, 30, 32, 33, 35,
36, 37, 39, 40, 41, 44, 45, 47, 48, 50, 51, 186, 45, 41, 64,
28, 30, 138, 238, 134, 168, 90, 37, 40, 410, 180, 348, 32, 25,
49, 49, 145, 41, 41, 40, 46, 30, 118, 204, 54, 201, 155, 62,
14, 528, 189, 185, 14, 6, 56, 52, 326, 96, 73, 284, 87, 70, 270,
327, 159, 389, 179, 103, 117, 648, 478, 641, 60, 42, 71, 100,
182, 100, 61, 187, 94, 72, 273, 282, 61, 422, 245, 169, 70, 783,
382, 326, 27, 4, 87, 105), .Dim = c(40L, 3L), .Dimnames = list(
NULL, c("ClinicID", "Flushots", "Uniquepatients")))


I want to see if the flu vaccine awareness campaign was effective or not (but I am interested in a two-sided test). Should I use a paired t-test on the differences in proportions? Wilcoxon Signed-rank test? What else (different techniques, concerns...) should I be thinking about?

• You cannot test this hypothesis about campaign effectiveness, because you have no controls for comparison. All you can hope to do is demonstrate there has been a change in clinic attendance and/or rates of vaccination between the two years. Any change, should you decide it is real, could be attributed to literally anything.
– whuber
Commented Apr 17, 2018 at 20:40
• Thank you. I understand that I cannot make a causal conclusion (observational study, no controls). The best I can show is that there has been a change and that it is real, as you put it. Do you think I can use a paired t-test or Signed-rank to answer whether this is real? Generally, what is the best approach to answer this types of question. Another example, all providers in 50 clinics are made to undergo a antibiotic stewardship training and we compare proportions of antibiotics prescribed for the clinics in a time window in 2016 vs 2017. Similar data of paired proportions for clinics. Commented Apr 17, 2018 at 20:55
• It depends on whether you need to make decisions per clinic or just overall. In either case, paired t-tests (which aren't too bad as an exploratory tool) probably should be adjusted for multiple comparisons.
– whuber
Commented Apr 17, 2018 at 21:57
• Thanks very much. I can understand doing paired t-tests for overall (independence of differences of post-pre; distribution probably appropriate).I agree multiple comparison adjustment is needed for clinic-specific decisions; I wouldn't think we could use paired t-test for this (comparing two proportions). I can accept your comment if you want to make it an answer regarding overall decision and if you have any insight into comparing proportions for clinic-specific decisions where the denominators are not the same and the samples are paired (same location), it would be helpful. Commented Apr 17, 2018 at 22:26
• Some variant of the answer by @GreggH would be a pretty good solution.
– whuber
Commented Apr 17, 2018 at 22:44

My recommendation would be a multi-level logistic multiple regression model where the dependent variable is having obtained the shot or not, the independent variable would be pre/post (dummy coded as 0/1), and a grouping variable for the clinics. (I am assuming you do not have patient identifying data to match the same clinic samples from pre to post.)

Follow-up from comments with code to run paired $t$-test and multilevel logistic regression with time as the independent variable.

library(lme4)

dat <- as.data.frame(structure(c(10, 20, 22, 24, 28, 30, 32, 33, 35, 36,
37, 39, 40, 41, 44, 45, 47, 48, 50, 51,
10, 20, 22, 24, 28, 30, 32, 33, 35, 36,
37, 39, 40, 41, 44, 45, 47, 48, 50, 51,
186,  45,  41,  64,  28,  30, 138, 238, 134, 168,
90,  37,  40, 410, 180, 348,  32,  25,  49,  49,
145,  41,  41,  40,  46,  30, 118, 204,  54, 201,
155,  62,  14, 528, 189, 185,  14,   6,  56,  52,
326,  96,  73, 284,  87,  70, 270, 327, 159, 389,
179, 103, 117, 648, 478, 641,  60,  42,  71, 100,
182, 100,  61, 187,  94,  72, 273, 282,  61, 422,
245, 169,  70, 783, 382, 326,  27,   4,  87, 105),
.Dim = c(40L, 3L),
.Dimnames = list( NULL, c("ClinicID", "Flushots", "Uniquepatients"))) )

## add time variable to data
dat <- cbind(dat, prop=dat$Flushots/dat$Uniquepatients, time=0)
dat$time[21:40] <- 1 ## drop one of the clinics trim.dat <- dat[which(dat$ClinicID != 48),]

## reshape the data for glm mixed effects analaysis
long.dat <- array(0,dim=c(sum(trim.dat$Uniquepatients),3)) cur.ind &lt;- 0 for(i in 1:38) { tmp.inds &lt;- cur.ind + 1:trim.dat$Flushots[i]
long.dat[tmp.inds,1] &lt;- trim.dat$ClinicID[i] long.dat[tmp.inds,2] &lt;- trim.dat$time[i]
long.dat[tmp.inds,3] &lt;- 1
cur.ind &lt;- cur.ind + trim.dat$Flushots[i] tmp.inds &lt;- cur.ind + 1:{trim.dat$Uniquepatients[i] - trim.dat$Flushots[i]} long.dat[tmp.inds,1] &lt;- trim.dat$ClinicID[i]
long.dat[tmp.inds,2] &lt;- trim.dat$time[i] long.dat[tmp.inds,3] &lt;- 0 cur.ind &lt;- cur.ind + trim.dat$Uniquepatients[i] - trim.dat$Flushots[i] } long.dat <- as.data.frame(long.dat) names(long.dat) <- c("Clinic","time","flu") ## run the data as a paired sample t-test t.test(dat$prop[c(1:17,19,20)],dat$prop[c(1:17,19,20)+20],paired=TRUE) ## run the data as a glm mixed effects model m <- glmer(flu ~ time + (1 | Clinic), data=long.dat, family = binomial) summary(m)  The$P$-value for the paired$t$-test is$p=.177$and the$P$-value for the time effect for the mixed effects model is$p<.001$. Output for paired$t$-test:  Paired t-test data: dat$prop[c(1:17, 19, 20)] and dat$prop[c(1:17, 19, 20) + 20] t = -1.4048, df = 18, p-value = 0.1771 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -0.07219102 0.01433476 sample estimates: mean of the differences -0.02892813  Output for multilevel logistic regression: Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod'] Family: binomial ( logit ) Formula: flu ~ time + (1 | Clinic) Data: long.dat AIC BIC logLik deviance df.resid 10997.3 11018.5 -5495.7 10991.3 8403 Scaled residuals: Min 1Q Median 3Q Max -2.4295 -0.9439 0.5903 0.8511 1.9137 Random effects: Groups Name Variance Std.Dev. Clinic (Intercept) 0.4441 0.6664 Number of obs: 8406, groups: Clinic, 19 Fixed effects: Estimate Std. Error z value Pr(>|z|) (Intercept) -0.003488 0.157582 -0.022 0.982341 time 0.177941 0.046336 3.840 0.000123 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Correlation of Fixed Effects: (Intr) time -0.136  • Thanks but I don't follow. I am not interested in whether individual patients were more likely to get the shot overall but rather whether there was a change in the number of vaccines administered overall in a hypothesis test framework. If I want to compare between two clinics in the same year, I can use a two-sample z-test for proportions. My issue is how to compare paired proportions for 20 clinics (pre/post percentage). In an ideal world, I would see that there is a difference between the mean paired difference as compared to 0 from the test and an increase based on the sample. Commented Apr 17, 2018 at 18:37 • This model would answer such a question. If the coefficient for the pre/post dummy variable is statistical significant, this suggests that there is a difference (aggregated across all clinics) between the likelihood of getting the flu shot before vs after the information campaign. Commented Apr 17, 2018 at 18:52 • Thank you. I will certainly look into that option but I am curious why you recommend this rather than a simpler hypothesis test on the paired differences in proportions. Independence between the post-pre measurements is satisfied as these are different clinics and it seems reasonable to assume that the difference in proportions is continuous (for a paired t-test) or at the very least something that can be dealt with by a Wilcoxon signed-rank or Sign test. Commented Apr 17, 2018 at 19:53 • Assuming the answer to my comment above is a matched-pairs t-test with 20 pairs of data, ¿what is the range of the proportions? If all of the sample sizes are high, and there aren't any extreme percentages, this may be a workable solution. However, it definitely depends on the$n$s and$p\$s. Commented Apr 17, 2018 at 22:58
• Thanks. Yes, I thought to use the number of vaccines administered divided by the number of patients it was offered to in 2016 and the number of vaccines administered divided by the number of patients it was offered to in 2017 as a matched pair of proportions; 2016 is pre and 2017 is post. It is matched because it is the same clinic and the importance of offering vaccines was stressed in the awareness campaign to the providers in each clinic. The ranges of the proportions is quite wide, almost 0.70 but nothing is too close to 0 or 1. Minimum n would be 40 and maximum 600. Commented Apr 18, 2018 at 3:01