1
$\begingroup$

Here is an example of a problem I am trying to figure out: I have data on 20 clinical locations where we recorded how many individuals came for a visit and how many of them chose to get the flu vaccine. Basically, for each of 20 locations, I have proportions. Then, following a flu vaccine awareness campaign, I have data for the same clinical locations again the following year. The total number of patients differs for each clinic between the two years and different clinics are also of different sizes (in terms of patients they saw during that window). So, data looks like:

Clinic 1 (pre/post) 1000/2000 and 1200/1900.
Clinic 2 (pre/post) 450/800 and 500/820.
... Clinic 20 (pre/post)...

Here are some example data in R:

dat <- structure(c(10, 20, 22, 24, 28, 30, 32, 33, 35, 36, 37, 39, 40, 
41, 44, 45, 47, 48, 50, 51, 10, 20, 22, 24, 28, 30, 32, 33, 35, 
36, 37, 39, 40, 41, 44, 45, 47, 48, 50, 51, 186, 45, 41, 64, 
28, 30, 138, 238, 134, 168, 90, 37, 40, 410, 180, 348, 32, 25, 
49, 49, 145, 41, 41, 40, 46, 30, 118, 204, 54, 201, 155, 62, 
14, 528, 189, 185, 14, 6, 56, 52, 326, 96, 73, 284, 87, 70, 270, 
327, 159, 389, 179, 103, 117, 648, 478, 641, 60, 42, 71, 100, 
182, 100, 61, 187, 94, 72, 273, 282, 61, 422, 245, 169, 70, 783, 
382, 326, 27, 4, 87, 105), .Dim = c(40L, 3L), .Dimnames = list(
    NULL, c("ClinicID", "Flushots", "Uniquepatients")))

I want to see if the flu vaccine awareness campaign was effective or not (but I am interested in a two-sided test). Should I use a paired t-test on the differences in proportions? Wilcoxon Signed-rank test? What else (different techniques, concerns...) should I be thinking about?

$\endgroup$
6
  • $\begingroup$ You cannot test this hypothesis about campaign effectiveness, because you have no controls for comparison. All you can hope to do is demonstrate there has been a change in clinic attendance and/or rates of vaccination between the two years. Any change, should you decide it is real, could be attributed to literally anything. $\endgroup$
    – whuber
    Commented Apr 17, 2018 at 20:40
  • $\begingroup$ Thank you. I understand that I cannot make a causal conclusion (observational study, no controls). The best I can show is that there has been a change and that it is real, as you put it. Do you think I can use a paired t-test or Signed-rank to answer whether this is real? Generally, what is the best approach to answer this types of question. Another example, all providers in 50 clinics are made to undergo a antibiotic stewardship training and we compare proportions of antibiotics prescribed for the clinics in a time window in 2016 vs 2017. Similar data of paired proportions for clinics. $\endgroup$
    – CVtest20
    Commented Apr 17, 2018 at 20:55
  • $\begingroup$ It depends on whether you need to make decisions per clinic or just overall. In either case, paired t-tests (which aren't too bad as an exploratory tool) probably should be adjusted for multiple comparisons. $\endgroup$
    – whuber
    Commented Apr 17, 2018 at 21:57
  • $\begingroup$ Thanks very much. I can understand doing paired t-tests for overall (independence of differences of post-pre; distribution probably appropriate).I agree multiple comparison adjustment is needed for clinic-specific decisions; I wouldn't think we could use paired t-test for this (comparing two proportions). I can accept your comment if you want to make it an answer regarding overall decision and if you have any insight into comparing proportions for clinic-specific decisions where the denominators are not the same and the samples are paired (same location), it would be helpful. $\endgroup$
    – CVtest20
    Commented Apr 17, 2018 at 22:26
  • $\begingroup$ Some variant of the answer by @GreggH would be a pretty good solution. $\endgroup$
    – whuber
    Commented Apr 17, 2018 at 22:44

1 Answer 1

1
$\begingroup$

My recommendation would be a multi-level logistic multiple regression model where the dependent variable is having obtained the shot or not, the independent variable would be pre/post (dummy coded as 0/1), and a grouping variable for the clinics. (I am assuming you do not have patient identifying data to match the same clinic samples from pre to post.)

Addendum #1
Follow-up from comments with code to run paired $t$-test and multilevel logistic regression with time as the independent variable.

library(lme4)

## read in data
dat <- as.data.frame(structure(c(10, 20, 22, 24, 28, 30, 32, 33, 35, 36,
                                 37, 39, 40, 41, 44, 45, 47, 48, 50, 51,
                                 10, 20, 22, 24, 28, 30, 32, 33, 35, 36,
                                 37, 39, 40, 41, 44, 45, 47, 48, 50, 51,
                                 186,  45,  41,  64,  28,  30, 138, 238, 134, 168,
                                  90,  37,  40, 410, 180, 348,  32,  25,  49,  49,
                                 145,  41,  41,  40,  46,  30, 118, 204,  54, 201,
                                 155,  62,  14, 528, 189, 185,  14,   6,  56,  52,
                                 326,  96,  73, 284,  87,  70, 270, 327, 159, 389,
                                 179, 103, 117, 648, 478, 641,  60,  42,  71, 100,
                                 182, 100,  61, 187,  94,  72, 273, 282,  61, 422,
                                 245, 169,  70, 783, 382, 326,  27,   4,  87, 105),
                               .Dim = c(40L, 3L),
                               .Dimnames = list( NULL, c("ClinicID", "Flushots", "Uniquepatients"))) )

## add time variable to data
dat <- cbind(dat, prop=dat$Flushots/dat$Uniquepatients, time=0)
dat$time[21:40] <- 1

## drop one of the clinics
trim.dat <- dat[which(dat$ClinicID != 48),]

## reshape the data for glm mixed effects analaysis
long.dat <- array(0,dim=c(sum(trim.dat$Uniquepatients),3))
cur.ind &lt;- 0
for(i in 1:38) {
   tmp.inds &lt;- cur.ind + 1:trim.dat$Flushots[i]
   long.dat[tmp.inds,1] &lt;- trim.dat$ClinicID[i]
   long.dat[tmp.inds,2] &lt;- trim.dat$time[i]
   long.dat[tmp.inds,3] &lt;- 1
   cur.ind &lt;- cur.ind + trim.dat$Flushots[i]
   tmp.inds &lt;- cur.ind + 1:{trim.dat$Uniquepatients[i] - trim.dat$Flushots[i]}
   long.dat[tmp.inds,1] &lt;- trim.dat$ClinicID[i]
   long.dat[tmp.inds,2] &lt;- trim.dat$time[i]
   long.dat[tmp.inds,3] &lt;- 0
   cur.ind &lt;- cur.ind + trim.dat$Uniquepatients[i] - trim.dat$Flushots[i]
   }
long.dat <- as.data.frame(long.dat)
names(long.dat) <- c("Clinic","time","flu")

## run the data as a paired sample t-test
t.test(dat$prop[c(1:17,19,20)],dat$prop[c(1:17,19,20)+20],paired=TRUE)

## run the data as a glm mixed effects model
m <- glmer(flu ~ time + (1 | Clinic), data=long.dat, family = binomial)
summary(m)

The $P$-value for the paired $t$-test is $p=.177$ and the $P$-value for the time effect for the mixed effects model is $p<.001$.

Output for paired $t$-test:

    Paired t-test

data:  dat$prop[c(1:17, 19, 20)] and dat$prop[c(1:17, 19, 20) + 20]
t = -1.4048, df = 18, p-value = 0.1771
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.07219102  0.01433476
sample estimates:
mean of the differences 
            -0.02892813 

Output for multilevel logistic regression:

Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod']
 Family: binomial  ( logit )
Formula: flu ~ time + (1 | Clinic)
   Data: long.dat

     AIC      BIC   logLik deviance df.resid 
 10997.3  11018.5  -5495.7  10991.3     8403 

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-2.4295 -0.9439  0.5903  0.8511  1.9137 

Random effects:
 Groups Name        Variance Std.Dev.
 Clinic (Intercept) 0.4441   0.6664  
Number of obs: 8406, groups:  Clinic, 19

Fixed effects:
             Estimate Std. Error z value Pr(>|z|)    
(Intercept) -0.003488   0.157582  -0.022 0.982341    
time         0.177941   0.046336   3.840 0.000123 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
     (Intr)
time -0.136
$\endgroup$
11
  • $\begingroup$ Thanks but I don't follow. I am not interested in whether individual patients were more likely to get the shot overall but rather whether there was a change in the number of vaccines administered overall in a hypothesis test framework. If I want to compare between two clinics in the same year, I can use a two-sample z-test for proportions. My issue is how to compare paired proportions for 20 clinics (pre/post percentage). In an ideal world, I would see that there is a difference between the mean paired difference as compared to 0 from the test and an increase based on the sample. $\endgroup$
    – CVtest20
    Commented Apr 17, 2018 at 18:37
  • 1
    $\begingroup$ This model would answer such a question. If the coefficient for the pre/post dummy variable is statistical significant, this suggests that there is a difference (aggregated across all clinics) between the likelihood of getting the flu shot before vs after the information campaign. $\endgroup$
    – Gregg H
    Commented Apr 17, 2018 at 18:52
  • $\begingroup$ Thank you. I will certainly look into that option but I am curious why you recommend this rather than a simpler hypothesis test on the paired differences in proportions. Independence between the post-pre measurements is satisfied as these are different clinics and it seems reasonable to assume that the difference in proportions is continuous (for a paired t-test) or at the very least something that can be dealt with by a Wilcoxon signed-rank or Sign test. $\endgroup$
    – CVtest20
    Commented Apr 17, 2018 at 19:53
  • $\begingroup$ Assuming the answer to my comment above is a matched-pairs t-test with 20 pairs of data, ¿what is the range of the proportions? If all of the sample sizes are high, and there aren't any extreme percentages, this may be a workable solution. However, it definitely depends on the $n$s and $p$s. $\endgroup$
    – Gregg H
    Commented Apr 17, 2018 at 22:58
  • $\begingroup$ Thanks. Yes, I thought to use the number of vaccines administered divided by the number of patients it was offered to in 2016 and the number of vaccines administered divided by the number of patients it was offered to in 2017 as a matched pair of proportions; 2016 is pre and 2017 is post. It is matched because it is the same clinic and the importance of offering vaccines was stressed in the awareness campaign to the providers in each clinic. The ranges of the proportions is quite wide, almost 0.70 but nothing is too close to 0 or 1. Minimum n would be 40 and maximum 600. $\endgroup$
    – CVtest20
    Commented Apr 18, 2018 at 3:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.