Analyzing multiple dependent pairs of 2x2 contingency tables

Question

In advance, I ask forgiveness for my statistical naïvety.

Suppose I have experimental data from pre- and post-treatment, and this data is split according to two binary categories like so:

\begin{array} {r|c|c|l} pre-treatment, q\#1 & \text{accurate} & \text{inaccurate} &\\ \hline \text{"I was right."} & 2 & 9 & 11 \\ \hline \text{"I was wrong."} & 14 & 2 & 16 \\ \hline & 16 & 11 \end{array}

Think of this data as someone first answering a question (e.g., is the sky blue?) and then they are asked to predict whether or not they correctly answered the original question (e.g., did you answer the question correctly?). In the above example, 4 respondents answered the original question correctly, with 2 participants accurately stating that they were right, and 2 participants inaccurately predicting that they were wrong. 9 participants inaccurately predicted that they were right when they weren't, and 14 particpants accurately predicted that they were wrong in their answer to the question.

Treatment is then applied, and again, the participant is asked both is the sky blue? and then did you answer this question correctly? which yields a similar contingency table for post-treatment, q#1.

\begin{array} {r|c|c|l} post-treatment, q\#1 & \text{accurate} & \text{inaccurate} &\\ \hline \text{"I was right."} & 24 & 1 & 25\\ \hline \text{"I was wrong."} & 0 & 2 & 2\\ \hline &24&3 \end{array}

Main Question: How can I statistically measure for significant changes in the pre- and post- samples both per-question and for the pre- and post- test as a whole?

Possible answer to my own question: Searches turn up multiple results suggesting McNemar's test, but it seems as though that test is applicable only if I re-categorize the pre and post data into the same contingency table. In this case, if I am attempting to answer the question "Does treatment improve subjects' ability to accurately predict their responses as correct/incorrect?", then the accurate and inaccurate data is all I care about from pre-treatment to post-treatment and I should arrange my contingency table like so:

\begin{array} {r|c|c|l} q\#1 & \text{accurate-post} & \text{inaccurate-post} &\\ \hline \text{accurate-pre} & 14 & 2 & 16 \\ \hline \text{inaccurate-pre} & 10 & 1 & 11 \\ \hline & 24 & 3 \end{array}

And if this is appropriate, am I correct in stating that McNemar's test is in some way measuring the effect of the treatment on the accuracy of each subjects' assessment of their own response to each question? (Again, it might bear repeating that this re-categorization is not based around whether or not each subject got the original question right is the sky blue?, but rather that they accurately predicted whether they were right or wrong after answering the question.)

Unresolved bit of question: And then finally, how can these measurements (applied for each is the sky blue? question) be reasonably compared/combined to consider a multitude of questions?

Answer 1

I have seen the type of data in the McNemar table many times, but the data in your first table is new to me. I don't understand how you get from the first table to the second. If 17 individuals answered "correct" on the first test, and 12 of them answered that the had given the right answer (essentially answering "correct" again), then there are 12 individuals who answer "correct" on both question and there should thus be 12 individuals in the upper left cell in the McNemar table, not 14. And the other cell counts should also be the same, though diagonally inverted. Or am I missing something obvious?

I think the second table is more easily understood so I think you should present your data to that format.

I've been in a similar situation before so I'll try to come up with some ideas.

Consider the table for the McNemar test above. We'll start at the upper left cell and call it a, and the upper right cell is b. The lower left cell is c and the lower right cell is d. What the McNemar test does is to test the marginal cell frequencies, i.e. cells b and c, and see if there is a statistically significant difference between them. It ignores cells a and d entirely, so the McNemar test is really only saying if the errors are skewed in either direction and it doesn't say anything about the proportion of individuals who are consistent in their answers (cells a and d). I don't know if there is a way to compare the results of a McNemar test across different sets of questions, as in your case, so I think you'll have to decide in what you're interested in.

You seem to be mostly interested in whether a certain question has a higher rate of agreement than other questions. You could then code it so that each individual has one row of response per question and a binary variable indicating if the individual was right in their assessment of their answer being correct. I mean, if they answered the same on both tests they should get a 1 and a 0 otherwise (we'll call this agreement). And you can then have a variable (correct) with 1 if the individual gave a correct answer the first time and 0 if the individual gave an incorrect answer:

subject question agree correct
1       1        1     1  
2       1        1     0  
3       1        0     0  
4       1        1     1  
5       1        0     1

You can now use a logistic regression here, to see if correctness influences the probability of agreeing (in your case above we might expect a significant positive effect of being correct: those who answer correctly seem more consistent in their results, i.e. more likely to "agree"):

subject <- seq(1:27)
question <- rep(1, 27)
agree <- c(rep(1, 12), rep(0, 9), rep(1,5), rep(0,1))
correct <- c(rep(1, 12), rep(0, 9), rep(0, 5), rep(1,1))
q1data <- data.frame(subject, question, agree, correct)
summary(glm(agree ~ correct, data=q1data))

Coefficients:
            Estimate Std. Error t value Pr(>|t|)   
(Intercept)   0.3571     0.1087   3.285  0.00302 **
correct       0.5659     0.1567   3.612  0.00133 **

This indicates that those who are correct have an odds ratio of exp(0.5659) = 1.8 for being in "agreement".

We can now do this again over more questions by using a generalized linear mixed model, and we then include question number as a factor and subject id as a random variable:

glmer (agree ~ correct + factor(question) + (1|subject), family=binomial)

And now you will model the probability of an individual being in "agreement" as a function of being correct and a function of question, with the multiple observations per individual taken into account by the random effect of subject. We'll create another dataset for question 2 here, using the counts from your McNemar table:

# Question 2
subject <- seq(1:27)
question <- rep(2, 27)
agree <- c(rep(1, 14), rep(0, 7), rep(1,3), rep(0,3))
correct <- c(rep(1, 14), rep(0, 7), rep(0, 3), rep(1,3))
q2data <- data.frame(subject, question, agree, correct)
qdata <- rbind(q1data, q2data)

And we now run the GLMM:

library(lmer)
summary(glmer(agree ~ correct + question + (1|subject), data=qdata, family=binomial))
Random effects:
 Groups  Name        Variance Std.Dev.
 subject (Intercept) 4.59     2.142   
Number of obs: 54, groups:  subject, 27

Fixed effects:
            Estimate Std. Error z value Pr(>|z|)   
(Intercept)   0.2599     1.5021   0.173   0.8627   
correct       3.9491     1.4720   2.683   0.0073 **
question     -0.8445     0.9519  -0.887   0.3750   

And now we can see that being correct is highly associated with being in "agreement", but there is no difference in this association between question 1 and question 2.

I hope this helps.

EDIT:

The problem in the question was made clearer. So we have two tests (pre- and post) and two assessments of this test. If you're interested in the agreement between the two assessments you could create one variable for pre-test [pre_agree] that is 1 if there is agreement between the assessment and the test, and 0 otherwise, and one variable for post-test [post_agree] that is 1 if there is agreement between the assessment and the test, and 0 otherwise. You can now model post_agree as an effect of pre_agree, pre_test (indicating right or wrong answer on pre-test), post_test (right/wrong on the post tets), perhaps their interaction, and other variables:

glmer (post_agree ~ pre_agree + pre_test + post_test + factor(question) + (1|subject), family=binomial)

An alternative is to include question as a random effect instead, if you're not interested in the effect of specific questions:

question <- factor(question)
glmer (post_agree ~ pre_agree + pre_test + post_test + (1|question) + (1|subject), family=binomial)