# Minimum number of trials per participant to assess within-group variability

I have a design in which participants can look at a correct or wrong picture (nominal dependent variable) as a function of between-subject age group (3 levels) and within-subject task type (2 levels).

When plotting the data, I have calculated the proportion of looking at the correct picture (y axis) in each group (e.g., age_1 - task_1) ACROSS PARTICIPANTS.

I did it across participants because each participant has only done 2-3 trials in each task condition. I would like to show mean proportion with standard errors but I don't know how meaningful these would be considering the I would have to calculate a proportion of correct responses per participant over 2-3 trials only, to then show the average across participants with a measure of deviation.

Do you think proportion of correct look per participant can still be used with this small number of trials?

My concern relates to the fact that with only 2 or 3 trials per participant, the proportion of correct look will be represented by a limited range of values (0 [0/2, 0/3]; 0.33 [1/3]; 0.5 [1/2]; 0.66 [2/3]; 1 [2/2, 3/3])

If you have a decent number of subjects, it's fine to show the mean of the individual proportions of correct responses (correct looks). In one sense, it's better than taking the proportion of correct responses while ignoring the clustering by patient. You essentially have individual estimates of the probability of a correct response, and you are averaging them for a group estimate of the probability of a correct response.

The challenge is your individual proportions of correct responses are not independently and identically distributed from some distribution. Some are being drawn from a distribution with a span of {0, 1/2, 1} and some with {0, 1/3, 2/3, 3}. You could argue they are iid from some mixture distribution, but either way, they aren't iid Normal. So if you were worried your standard t confidence interval (CI) for the mean which uses a critical value from the t-distribution wouldn't perform well in your setting, you could write a small simulation study to check its coverage. I did this below. Alternatively, you could compare those t CIs to the CIs from a good bootstrap approach.

The performance of the t interval will depend on your number of subjects, number of trials per subject, and true probability of a correct look. You'll want to tweak the code to match your situation better. The simulation below gets 93% coverage - which isn't terrible. HOWEVER, change theta to 0.9 and coverage drops to 89% - which ain't great if you're wanting a 95% CI. So the validity of the t CI will depend on your study's settings. I'm allowing some subjects to have only 1 trial (see NsDist below). When the number of trials is always 2 or 3, performance is a little better, but it still depends on nsubs and theta.

## R script to validate my gut sense
loops <- 10^5
coverage <- rep(NA,loops)
# distribution of indivual sample sizes
NsDist <- c(1,2,2,2,3,3)
# true probability of a correct "look"
theta <- 0.7
# number of subjects
nsubs <- 10
# estimate coverage
for(i in 1:loops){
# generate one sample of individual proportions
props <- rep(NA,nsubs)
for(j in 1:nsubs){
n <- sample(NsDist,1)
props[j] <- rbinom(1,n,theta)/n
}
# calculate CI; if no variation set CI bounds to mean
if(min(props)==max(props)){ ci <- c(mean(props),mean(props))
} else { ci <- t.test(props)\$conf.int }
# check coverage
coverage[i] <- (ci[1] < theta & theta < ci[2])
}
# coverage estimate (as %)
round( 100*mean(coverage), 1 )

• Thanks Robert! I tried the simulation for each group setting theta according to previous literature. The coverage is always above 94% as I have 27-34 participants in each group, theta between .3 and .8, and always 2-3 individual sample sizes. I have a doubt. In my design there is a within-subject variable. Do you think I could do the simulation applying Morey (2008) correction (bottom of this page tiny.cc/04n5fz) to control for inter-subject variability when simulating the t CIs? So that I would have a coverage measure that justifies the use of corrected t CIs? Nov 12, 2019 at 13:30
• Correction CI paper is also available here: pcl.missouri.edu/sites/default/files/morey.2008.pdf Nov 12, 2019 at 15:03
• Thanks for the link to the paper. I'm hesitant to sign off on either the Loftus and Masson (1994) or Cousineau (2005) in this particular scenario, because if I followed the paper correctly, both call upon a pooled estimate of the variance. In your setting, there is a mean-variance association, i.e. groups with theta ~0.5 will have a larger variance than groups with theta ~0.8. If trying a more advanced approach, I would consider going back to the data with 2-3 rows for each subject and fit a mixed effects logistic regression of "correct look" by age and task with a random effect for subject. Nov 12, 2019 at 18:20
• Thanks very much for your answer Robert! I already planned to use a mixed effects model on the data and thanks for confirming that. I wasn't sure on how to plot the data first though, but you have clarified my doubts so I will use the t CIs then for a mere graphical purpose! Nov 12, 2019 at 20:08