Matched binomial pairs

Question

I am trying to analyse the following table:

5 5 5 5 4 5 5 5 5

3 4 3 3 4 4 3 5 5

Each number is a success score out of 5 attempts, so we can assume binomial (p, 5), for unknown p. Each column pertains to one person. The bottom row contains the "pre" test results and the top row is the "post" test results.

At a quick glance, every item in the top row is at least as large as the corresponding item in the bottom row, so by a quick sign test, the hypothesis that there is a difference pre and post should be significant.

If I do a paired t-test on these data (holding my nose, because each item is binomial (p,5)) I get a significant result. If I put the data into a 2 x 2 contingency table (pre post x right wrong) and ignore the matching, I get a significant result.

But when I do a contingency test of the table above, testing for independence conditional on the margins, the result is non-significant (p = 0.97). I used fisher.test in R, the Fisher exact test.

So what is the best way to test the hypothesis that there is a significant difference, pre and post, taking matching into account? And why does the analysis of independence not apply here?

Answer 1

I would recommend you to use Generalized Estimating Equations (GEE). This is analogous to Repeated-measures ANOVA but allows for non-continuous or non-normal response because, being a Generalized linear model, it adopts various link functions.

With your example, I'd use binomial distribution with logit link (albeit you might prefer probit). The input data was restructured from "wide" into "long" which appears this:

   id pre_post count
    1   1       5
    2   1       5
    3   1       5
    4   1       5
    5   1       4
    6   1       5
    7   1       5
    8   1       5
    9   1       5
    1   2       3
    2   2       4
    3   2       3
    4   2       3
    5   2       4
    6   2       4
    7   2       3
    8   2       5
    9   2       5

The DV is count, how much a respondent scored out of 5 attempts. pre_post is the repeated-measures factor levels of which you want to compare. Id is respondent. I'll use SPSS to analyse it; the command is below.

GENLIN count /*response variable: score
  OF 5 /*out of 5 trials
  BY pre_post
 /MODEL pre_post INTERCEPT=YES /*the only factor to test is pre_post
  DISTRIBUTION=BINOMIAL LINK=LOGIT
 /REPEATED SUBJECT=id WITHINSUBJECT=pre_post /*id is respondent id, pre_post is the repeated measures factor
  CORRTYPE=INDEPENDENT ADJUSTCORR=YES /*we'll assume Independent correlation structure.

Here is an excerpt from the results:

You see that is a significant difference between pre and post. Usual RM ANOVA (paired-samples t-test) gave close significance, .007.

Answer 2

I don't think fisher exact test is appropriate here is it?

It is saying that pre-test there are 34 points arranged in this manner (343344355), and that post test there are 44 points. The arrangement of those 44 points is consistent with the pattern (marginal) established in the pre-test. It is also saying that each pairing looks similar (row part of the test) - people generally do better in the second test by the same amount.

Fisher test isn't testing that the rows are equal, just that they are independent of the columns. You want to answer the question "Do people do better post-test", but fisher test is asking the question "Do all people do better on the post-test in the same way".

I think sign test is perfectly appropriate here, and I've just see @ttnphns answer appear which is particularly good.

Answer 3

One other option would be run a binomial mixed model. You would need to code the data like this:

result  id  time
 0       1   pre
 0       1   pre
 1       1   pre
 1       1   pre
 1       1   pre
 1       1   post
 1       1   post
 1       1   post
 1       1   post
 1       1   post

With the R package lme4 you could then run the modell

lmer(result ~ timepoint + (1|id), family = "binomial")

But with results like these I suspect any method will deliver significant results and it is properly convenient if you use a method with sound theoretical background with which you are comfortable AND yields the answer you want.

The sign test is absolutely sufficient and appropiate if you just want to show an improvement, but when you also want to quantify the improvement and variation between individuals you start needing more complex models.