Let's say I have data from an experiment:
- 3 participants took part
- each of them was subject to two conditions. Condition A, and B.
- in Condition A, 3 trials could yield success or failure (depending on the performance)
- in Condition B, a variable number of trials could yield success or failure (also depending on the performance). Note: different participants have different amounts of trials.
The experiment is now conducted and as a check, I want to see:
- whether participants overall got more successes than failures.
whether the proportion of successes differed between the two conditions A and B.
# Example code in R subj <- c(1,1,1,1,1,1,1,1,2,2,2,2,2,2,3,3,3,3,3,3,3,3,3) condition <- c(rep('a',3), rep('b',5), rep('a',3), rep('b',3),rep('a',3),rep('b',6)) outcome <- c(1,1,0,1,0,1,1,0,1,1,1,0,1,1,1,0,1,1,1,0,1,1,1) df_long <- data.frame(subj, condition, outcome) ggplot(df_long, aes(as.factor(outcome), fill=condition)) + geom_bar()
During the last hours, I have read about several tests (z-test, fisher's exact test, chi-square, binomial test, mcNemar...) and these are my thoughts:
To answer the first question, whether participants overall got more successes than failures, I could do a binomial test, comparing the observed proportion of successes (collapsed over conditions) to a theoretical quantity of 0.5. If in a one-tailed test, I get a significant result, would it mean that successes were significantly bigger than 50% ... and in reverse that would mean that failures were smaller than 50% ... and thus successes>failures? --> Is there a better way to approach this? It feels a bit like cheating somehow, because I do not account for how the count of overall successes is "produced" by an interplay of conditions A and B.
To answer whether proportions of successes differ between conditions A and B, a McNemar's Test might have been appropriate, because:
- This is paired nominal data with a "dichotomous trait"
- HOWEVER ... this data is not matched, because I have 3 trials for condition A, but variable amounts of trials for condition B. I wouldn't even know how to construct a contingency table for this.
- Does this perhaps mean, that I should not treat my data as paired?
- Which test is appropriate here?
EDIT: The way the data is produced actually is not a binomial process, because neither are the single trials in the sequence independent, nor do they share the same probability p to yield a success.
Looking at it from a GLMM perspective like gung suggested in the comment might be a better idea. Does anyone know what test to apply if the number of data instances in the second class differ for each participant?