How to test for significant differences in means across four within subject conditions where dependent variable is a proportion?

Question

I have a dataset with four within-subject conditions. Participants gave binary answers to 8 questions in each condition, and these were averaged to form a proportion for each condition. I would like to compare the mean proportions of people choosing a particular answer between conditions. For example 80% chose a particular answer in the first condition, 60% in the second 40% in the third 20% in the fourth.

Here's a graph of the results: Mean probability of guessing RED drops as no. GREEN increases. I want to see whether the drop is significant.

• How can I somehow summarize whether this drop is significant?

Answer 1

Here's a simple rule: when you can compute a proportion within an individual, don't. Dixon (2008) and Jaeger (2008) both demonstrate that this can lead to erroneous inferences. The proper approach to analysis of repeated binary data is to use an inferential approach that treats the data as binary. Here is code (for R) to grab the latest version of the ez package and compute likelihood ratios for your design's effects (and, by the way, treating your numeric variables as continuous but possibly non-linear via gam, thereby enhancing power):

#install CRAN ez
install.packages('ez')
library(ez)

#get ready to retrieve Dev version of ez
install.packages('RCurl')

#retrieve ezDev
source('https://raw.github.com/mike-lawrence/ez/master/R/ezDev.R')

#load Dev version of ez's functions into memory
ezDev()

#now run the model
my_mix = ezMixed(
    data = my_data
    , dv = .(choice_is_red)
    , random = .(participant)
    , fixed = .(num_green,num_red,message)
)
print(my_mix$summary)
#In the summary, the bits column represents the computed evidence 
#associated with each effect, on the log-base-2 (aka "bits") scale.
#The absolute value represents the strength of evidence while the sign 
#represents whether the effect (+) or its null (-) is supported.

#visualize the 3-way with CIs that eliminate between-participants variance
preds = ezPredict(
    fit = my_mix$models$'num_green:num_red:message'$unrestricted
)
p = ezPlot2(
    predictions = preds
    , x = .(num_green)
    , split = .(num_red)
    , row = .(message)
    , x_lab = 'No. green'
    , split_lab = 'No. red'
    , y_lab = 'Likelihood of choosing red (log-odds)'
)
print(p$plot)

This code assumes that your data is stored in the object my_data, which has the following structure (order of columns is unimportant, just that they're all there and that it is the raw question-by-question info for each participant):

participant question num_red num_green message choice_is_red
sub1        1        3       2         red     0
sub1        2        4       4         red     1
sub1        3        1       3         blue    0
...
sub2        1        2       1         blue    0
sub2        2        1       2         red     1
...

Answer 2

[Note: The following reply pertains to older drafting of the question, "How to test for significant differences in proportions across four within subject conditions?" Later the question was cleared up in that the proportions are actually averaged across various variables.]

You might use McNemar's test which is a repeated-measures comparison of proportions. Classic form of the test is for binary response and is therefore suits you. The test is pairwise comparison: only two conditions at a time. The 2x2 frequency table (Yes, No one condition vs Yes, No another condition) is formed and the Ho that the table is symmetric about the diagonal is tested.

When variables are dichotomous (like yours) McNemar test is equivalent to Sign test so you could apply that either.

Check also Cochran Q test as an extension of McNemar's test from pairwise to omnibus comparison