Establish confidence / error for multiple contingency tables with dynamic outcomes

Question

In a task, a participant sees two types of prompts, 1 and 2. The participant might answer A or B. The participant might give an answer that cannot be coded as A or B, we treat these as missing data. The participant responds eight times.

Correct responses can be only ascertained on the participant level. If a participant responds A to prompt 1 and B to prompt 2 OR A to 2 and B to 1, the participant is consistent. Non-matching answers result in a loss of consistency.

This can be operationalised into a contingency table, where a perfectly consistent participant is

A B
0 4
4 0

or

A B
4 0
0 4

A less consistent participant might look like

A B
1 3
4 0

A terribly inconsistent participant will, of course, look like

A B
2 2
2 2

Given the low cell counts, I can use e.g. a Fisher exact test to get a p value about the participant's consistency. In this setup, a 4/0 0/4 participant will be p < 0.05, everybody else will be p > 0.05 and so on (with missing data, you might see 3/0 0/4 etc). I'm fine with this.

I can simulate participants to show how many would get p > 0.05 by chance.

How do I account for the number of participants in my sample? Surely, having 500/1000 participants showing consistent behaviour will be more informative / result in less error than having 5/10.

I tried to reduce the problem to a linear model as per this question but my problem is that I can't define a correct outcome on the data level, only on the participant level (since 1:A 2:B AND 1:B 2:A are both consistent, and deviations should be counted from the participant tables).

The question is: How do I account for the number of participants? I can determine the prob

Answer 1

What I ended up doing:

For each participant, establish the most likely answer for 1 and 2.

So if the table is

AB
40
04

then 1=A 2=B.

If the table is

AB
31
13

it's still 1=A 2=B.

If the table is

AB
31
31

it's 1=A 2=A.

If the chances are even, we pick something at random:

If the table is

AB
22
22

here both 1=A 2=B and 1=B 2=A are fine.

Code individual answers based on whether they match the coded majority "correct" category

So if, for participant i, 1=A and 2=B, then a response of A to 1 is correct, a response of B to 1 is incorrect, etc.

Fit binomial model

We can now ask whether log (correct/incorrect) > 0.