How can I analyze the effects of a three-level within-subjects factor on an outcome measure, when the effects at the three levels are not independent?
I ran an experiment in which each participant had to choose among four options (a, b, c, or d) on each of 12 trials, representing 4 trials for each of 3 levels (X, Y, and Z) of a within-subjects factor. I transformed responses into binary (0 or 1) format where 0 means "either a or b" and 1 means "neither a nor b". So, normally, I would think I could analyze my data with an ANOVA, with "proportion of trials in which response==1" as the dependent variable, and with my within-subjects factor as the main indendent variable of interest, as shown in the reproducible example below.
However, participants were constrained to choose a, b, c, and d equally often - i.e., 3 times each over 12 trials. As a result, each participant would have values of 0 and 1 on exactly 6 trials each. This constraint means that, for the different levels of the within-subjects factor, the proportions of trials in which response==1 are not independent. So, I think I can't use ANOVA. Am I correct?
If so, what should I do instead? Ideally I'd like something as close to ANOVA as possible, which would allow me to also test for effects of between-subjects factors, as shown in the example below.
library( plyr )
library( reshape2 )
library( ez )
# create data
D = expand.grid( grade=c("6","8"), problem_set=c("set 1","set 2"), subjid=1:30 )
D$subjid = paste( D$grade, D$subjid, sep="_" )
# randomly generate the proportion of trials in which response==1 for each level (X, Y, Z) of the within-subjects factor,
# satisfying the constraint that the average proportion of trials in which response==1 across these levels is 0.50
simulate_ss = function(r) {
v = rep( c("X","Y","Z"), each=4 )
w = sample( v, 6, replace=FALSE )
r$X = sum(w=="X")/12
r$Y = sum(w=="Y")/12
r$Z = sum(w=="Z")/12
return(r)
}
D = adply( D, 1, simulate_ss )
D = melt( D, measure.vars=c("X","Y","Z"), variable.name='within_var', value.name='percent' )
# analyze data
ezANOVA( data=D, wid=subjid, dv=percent, within=within_var, between=.(grade, problem_set) )
