# Help with choosing a statistical test: Possible repeated-measures ANOVA?

Apologies if this is a simple question; my knowledge is limited to a single introductory stats unit.

My design has two independent groups: test group vs control group. Both will be presented with content-based thought probes, in which the responses will be classified in three ways: on-task, task-related interference, or task-unrelated thoughts. I want to examine if the two groups differ in the proportions of thought types they report.

A similar study used a 2 (group: tested vs control) x3 (thought probe response) repeated-measured ANOVA to assess for group differences, followed by post-hoc two-tailed t-tests to determine where those differences lie. However, my readings on repeated-measures ANOVAs suggest that these are mainly used for within-group designs. Is a repeated-measures ANOVA the appropriate test for my study, and if not, what should I use?

Edit:

Research Aim: To explore whether pretests reduce mind wandering in online lectures.

DV: Rate of mind wandering, first measured using the mean proportion of probe responses to choices 4 & 5 (see below). Probes are presented at 4 time points throughout the lecture to both groups. I will use an independent samples t-test to explore this first question.

“What are you thinking about right now?”

1. what the professor is saying right now (on-task)
2. something related to an earlier part of the lecture (task-related interference)
3. relating the lecture to my own life (task-related interference)
4. something unrelated to the lecture (task-unrelated thoughts)

However, I wish to expand on this and further differentiate the thought types to assess if there are group differences in the mean proportions of other thought types (e.g., if the tested group report significantly different proportions of on-task thoughts (response 1) and/or task-related interference thoughts (response 2 & 3) etc. compared to the control group).

Related example: Jing, H. G., Szpunar, K. K., & Schacter, D. L.(2016). Interpolated Testing Influences Focused Attention and Improves Integration of Information During a Video-Recorded Lecture. Journal of Experimental Psychology: Applied, 22(3), 305–318. https://doi.org/10.1037/xap0000087 See: Experiment 2 Results: Thought probes and note taking.

• Please edit the question to say more about the study design and outcome variable, and to include a reference (with a web link, if possible) to the "virtually identical study" that you note. If outcomes are just the three categories on-task, task-related interference, or task-unrelated thoughts then ANOVA isn't correct as ANOVA assumes a continuous outcome. There are ways, however, to deal with categorical outcomes. Whether you need to do "repeated measures" analysis depends on how each individual is exposed to the "content-based thought probes" over time. Please explain that, too.
– EdM
Feb 10, 2023 at 13:54

The problem is that the variance of a proportion estimate $$p$$ depends on the estimate, specifically related to $$p(1-p)$$. That product is 0.25 for $$p=0.5$$, but it's only 0.09 for $$p=0.1$$ and about 0.02 for $$p=0.02$$. Offsetting that problem is that the large differences in variance go along with large differences in proportions being compared, so "significance" comparisons might end up good enough in practice. This page is a good place to start on those issues.
Do that first. That models each of the outcomes from all of your data at once, in a way that is closest to the original observations. Then you can use post-modeling tools like those in the emmeans package to evaluate whatever combinations of response values you would like between groups, translated from log-odds to proportion scales with appropriate confidence intervals, and with multiple-comparison corrections as needed.