# Paired Comparison on $n$ Tests, Repeated $k$ Times Each

Suppose I test two subjects on $$n$$ pass/fail (Bernoulli) tasks, $$k$$ times each—for a total of $$n\cdot k$$ tests per subject. I want to compare the subjects to see if there is a statistically significant difference in their ability to pass a random task. Many tests, such as the two-proportion z-test, are eliminated because both within the test results of subjects and between the subjects, observations are dependent. What is the most appropriate statistical test in this case?

(I think I am looking for something like McNemar-Bowker, but that doesn't seem quite right, unless I'm mistaken. What's weird is that there is not a one-to-one pairing between the results of each subject; it's more like "$$k$$-to-$$k$$ pairings, $$n$$ times each.'')

It would also be desirable to construct a confidence interval on the difference between the subjects, if possible.

• "both within the test results of subjects and between the subjects, observations are dependent. What is the most appropriate statistical test in this case?" Do you have a description of this dependence? Is it like a random model e.g. a person has some probability $p$ to get a successful task making the task results correlated within the person? And the between subject correlation, that is because tasks in different tests have different difficulty and easier test and difficult tests result in respectively high and low success for both subjects? Dec 11, 2023 at 9:09
• "to see if there is a statistically significant difference between them" A difference in what way? E.g. a difference in their means of all test scores, or a difference in their profile among the different tests? And what likelihood/probability function do you assume to test/express this? How do you consider the distribution of results given the null hypothesis 'no difference' is true? Dec 11, 2023 at 9:13
• If a subject passes/fails a given task once, it is likely to pass/fail it again. Many of the $n$ tasks are also extremely similar to each other. Furthermore, the subjects are fairly similar, meaning that if a task is easy/hard for one subject, it is likely to be easy/hard for the other. The subjects are tested on precisely the same tasks. You could say that for task $m,$ subject $i$ has a probability $p_{i,m}$ of passing, and there is (1) a high correlation between $p_{1,m}$ and $p_{2,m}$ for all $m$, and (2) a reasonable correlation between $p_{i,m}$ and $p_{j,m}$ for many pairs $i,j$. Dec 11, 2023 at 9:50
• “You could say that for task $m$, subject $i$ has a probability $p_{i,m}$ of passing, ” You are considering this like a Bernoulli distribution? So the outcome variables are considered independent variables given a value $p_{i,m}$? If that is true, then how do you consider the correlations? Is it like the values $p_{i,m,}$ themselves are following a distribution and are correlated? Dec 11, 2023 at 13:10
• Hi @kjetilbhalvorsen, I have accepted your answer. To be specific, I was evaluating LLMs against language tasks. They were asked $n$ simple reasoning questions $k$ times each, and they either passed or failed. My hope was to see if, upon changing the wording of the questions, but not the meaning, the LLMs performance changed. Remarkably, it did not significantly change under any of the standard-yet-less sophisticated significance tests, but I wanted to be sure. Mar 28 at 2:20

It seems like a mixed model is in order, for instance implemented via the lme4 package in R. Since the main interest is in comparing individuals, we make the individual effects as a fixed effect, and model the different tasks via random effects. The information given in the comments indicate that interaction via individuals and tasks is not expected (interactions would also be difficult to estimate). So within logistic regression we model the probabilities $$p_{im}$$ ($$i$$ indexing individuals, $$m$$ indexing tasks) as $$\operatorname{logit}(p_{im}) = \mu + \alpha_i + \beta_m$$ where $$\alpha_i$$ is fixed effects and $$\beta_m$$ are random variables with expectation zero. In R we can do something like

library(lme4)

mod0 <- lme4::glmer( response ~ ID + (1 | task),
family=binomial,
data=your_data_frame)
summary(mod0)


Consider a relatively simple procedure to test whether two subjects are different, or whether one is better than the other.

First summarize the response of each subject to each task by computing the number of times they pass, ranging from 0 to k. (This allows for the common correlation among responses to a task.)

You can then compare the scores of two subjects with a simple sign test, comparing the number of tasks that subject A has a higher (positive sign) or lower (negative sign) score than subject B has, which accounts for the tasks where the two subject tie. (You could use another test for paired data if it seemed more appropriate, eg, if there were very few ties.).

This gives you an overall test for a difference between subjects. It does not give you some of the detailed information that the first answer provides.

If some tasks are more complicated than others, you will see a lot of variation among the task scores, for either subject or for the total of the two subjects. If there is more variation among task scores, then it seems you are more likely to find a difference between subjects.

If there is a lot of variation among task scores and you find little difference between subjects then it is likely that your tasks are not evaluating a common skill or ability. You should review whether your tasks are useful.