test two binomial distributions: overall successes > failures and proportion successes A≠B Let's say I have data from an experiment:


*

*3 participants took part

*each of them was subject to two conditions. Condition A, and B.

*in Condition A, 3 trials could yield success or failure (depending on the performance)

*in Condition B, a variable number of trials could yield success or failure (also depending on the performance). Note: different participants have different amounts of trials.


The experiment is now conducted and as a check, I want to see:


*

*whether participants overall got more successes than failures. 

*whether the proportion of successes differed between the two conditions A and B.
# Example code in R
subj <- c(1,1,1,1,1,1,1,1,2,2,2,2,2,2,3,3,3,3,3,3,3,3,3)
condition <- c(rep('a',3), rep('b',5), rep('a',3), rep('b',3),rep('a',3),rep('b',6))
outcome <- c(1,1,0,1,0,1,1,0,1,1,1,0,1,1,1,0,1,1,1,0,1,1,1)
df_long <- data.frame(subj, condition, outcome)

 ggplot(df_long, aes(as.factor(outcome), fill=condition)) + geom_bar()


During the last hours, I have read about several tests (z-test, fisher's exact test, chi-square, binomial test, mcNemar...) and these are my thoughts:


*

*To answer the first question, whether participants overall got more successes than failures, I could do a binomial test, comparing the observed proportion of successes (collapsed over conditions) to a theoretical quantity of 0.5. If in a one-tailed test, I get a significant result, would it mean that successes were significantly bigger than 50% ... and in reverse that would mean that failures were smaller than 50% ... and thus successes>failures? --> Is there a better way to approach this? It feels a bit like cheating somehow, because I do not account for how the count of overall successes is "produced" by an interplay of conditions A and B.

*To answer whether proportions of successes differ between conditions A and B, a McNemar's Test might have been appropriate, because:


*

*This is paired nominal data with a "dichotomous trait"

*HOWEVER ... this data is not matched, because I have 3 trials for condition A, but variable amounts of trials for condition B. I wouldn't even know how to construct a contingency table for this.

*Does this perhaps mean, that I should not treat my data as paired?

*Which test is appropriate here?



EDIT: The way the data is produced actually is not a binomial process, because neither are the single trials in the sequence independent, nor do they share the same probability p to yield a success.
Looking at it from a GLMM perspective like gung suggested in the comment might be a better idea. Does anyone know what test to apply if the number of data instances in the second class differ for each participant?
 A: Okay, after some more research and discussions, I think I have a solution:
Reiterate data set
library(ggplot2)
library(dplyr) # we need this package now

# The initial data set
subject <- c(1,1,1,1,1,1,1,1,2,2,2,2,2,2,3,3,3,3,3,3,3,3,3)
condition <- c(rep('a',3), rep('b',5), rep('a',3), rep('b',3),rep('a',3),rep('b',6))
outcome <- c(1,1,0,1,0,1,1,0,1,1,1,0,1,1,1,0,1,1,1,0,1,1,1)
df_long <- data.frame(subject, condition, outcome)

View(df_long)

# The initial plot
ggplot(df_long, aes(as.factor(outcome), fill=condition)) + geom_bar()

Question 2:
We treat this test as a 2nd level analysis and take a subject as the unit of observation instead of the single successes of each subject. Like that, we can collapse the data over the single trials and end up with one proportion of successes per condition per participant.
With this type of data organization, we can then do a two sample paired t-test. Of course, we lose some information with this approach - namely the variance how the proportions for each participant come about - ... so if somebody has a better idea, let me know!
# Collapse the data across single trials. We will end up with
# one proportion of successes per condition per participant
collapseSubjCond <- df_long %>% group_by(subject, condition) %>% summarize(meanOutcome=mean(outcome))


# Make a more sensible plot showing just the proportion of correct.
# Showing the porportion of incorrect as well is kind of a duplicate because
# it is apparent from the proportion correct
dummy <- collapseSubjCond %>% group_by(condition) %>% summarize(meanOutcome=mean(meanOutcome))
ggplot(dummy, aes(x = condition, y = meanOutcome, fill = condition)) + geom_col()


# Make a paired t-test
t.test(meanOutcome~condition, data=collapseSubjCond, paired=TRUE)


Question 1:
With the same general idea as in Question 2, we treat the subject as a unit of observation and do a paired t-test between meanSuccesses and meanFailures per participant ... collapsed over conditions
# From our data frame that contains one proportion of successes per condition
# and per participant, we want only one proportion of successes per participant
dummy <- collapseSubjCond %>% group_by(subject) %>% summarize(meanOutcome=mean(meanOutcome))

# Now we have three participant. One proportion of successes for each participant
# and the failures are 1-successes
meanSuccess <- dummy$meanOutcome
meanFailure <- 1-meanSuccess

# simply do a paired t-test
t.test(meanSuccess, meanFailure, paired=TRUE)

