Help with choosing a statistical test: Possible repeated-measures ANOVA?

Question

Apologies if this is a simple question; my knowledge is limited to a single introductory stats unit.

My design has two independent groups: test group vs control group. Both will be presented with content-based thought probes, in which the responses will be classified in three ways: on-task, task-related interference, or task-unrelated thoughts. I want to examine if the two groups differ in the proportions of thought types they report.

A similar study used a 2 (group: tested vs control) x3 (thought probe response) repeated-measured ANOVA to assess for group differences, followed by post-hoc two-tailed t-tests to determine where those differences lie. However, my readings on repeated-measures ANOVAs suggest that these are mainly used for within-group designs. Is a repeated-measures ANOVA the appropriate test for my study, and if not, what should I use?

Edit:

Research Aim: To explore whether pretests reduce mind wandering in online lectures.

DV: Rate of mind wandering, first measured using the mean proportion of probe responses to choices 4 & 5 (see below). Probes are presented at 4 time points throughout the lecture to both groups. I will use an independent samples t-test to explore this first question.

“What are you thinking about right now?”

1. what the professor is saying right now (on-task)
2. something related to an earlier part of the lecture (task-related interference)
3. relating the lecture to my own life (task-related interference)
4. something unrelated to the lecture (task-unrelated thoughts)
5. zoning out (task-unrelated thoughts)

However, I wish to expand on this and further differentiate the thought types to assess if there are group differences in the mean proportions of other thought types (e.g., if the tested group report significantly different proportions of on-task thoughts (response 1) and/or task-related interference thoughts (response 2 & 3) etc. compared to the control group).

Related example: Jing, H. G., Szpunar, K. K., & Schacter, D. L.(2016). Interpolated Testing Influences Focused Attention and Improves Integration of Information During a Video-Recorded Lecture. Journal of Experimental Psychology: Applied, 22(3), 305–318. https://doi.org/10.1037/xap0000087 See: Experiment 2 Results: Thought probes and note taking.

Answer 1

I'm not sure that the linked paper would have survived statistical review on this website. Although t-tests and ANOVA can sometimes work well enough with proportion data, they can pose problems with the fundamental assumption of equal within-cell variances. It's also not clear, at a quick glance, how well the paper dealt with the multiple comparisons problem.

The problem is that the variance of a proportion estimate $p$ depends on the estimate, specifically related to $p(1-p)$. That product is 0.25 for $p=0.5$, but it's only 0.09 for $p=0.1$ and about 0.02 for $p=0.02$. Offsetting that problem is that the large differences in variance go along with large differences in proportions being compared, so "significance" comparisons might end up good enough in practice. This page is a good place to start on those issues.

As the responses seem to be 5-alternative forced-choice, multinomial regression might be a better choice. You would model the log-odds of each of options 2 through 5 against on-task option 1 (which seems to be the most prevalent response in the linked study), as functions of experimental conditions.

Do that first. That models each of the outcomes from all of your data at once, in a way that is closest to the original observations. Then you can use post-modeling tools like those in the emmeans package to evaluate whatever combinations of response values you would like between groups, translated from log-odds to proportion scales with appropriate confidence intervals, and with multiple-comparison corrections as needed.