Apologies if this is a simple question; my knowledge is limited to a single introductory stats unit.
My design has two independent groups: test group vs control group. Both will be presented with content-based thought probes, in which the responses will be classified in three ways: on-task, task-related interference, or task-unrelated thoughts. I want to examine if the two groups differ in the proportions of thought types they report.
A similar study used a 2 (group: tested vs control) x3 (thought probe response) repeated-measured ANOVA to assess for group differences, followed by post-hoc two-tailed t-tests to determine where those differences lie. However, my readings on repeated-measures ANOVAs suggest that these are mainly used for within-group designs. Is a repeated-measures ANOVA the appropriate test for my study, and if not, what should I use?
Research Aim: To explore whether pretests reduce mind wandering in online lectures.
DV: Rate of mind wandering, first measured using the mean proportion of probe responses to choices 4 & 5 (see below). Probes are presented at 4 time points throughout the lecture to both groups. I will use an independent samples t-test to explore this first question.
“What are you thinking about right now?”
- what the professor is saying right now (on-task)
- something related to an earlier part of the lecture (task-related interference)
- relating the lecture to my own life (task-related interference)
- something unrelated to the lecture (task-unrelated thoughts)
- zoning out (task-unrelated thoughts)
However, I wish to expand on this and further differentiate the thought types to assess if there are group differences in the mean proportions of other thought types (e.g., if the tested group report significantly different proportions of on-task thoughts (response 1) and/or task-related interference thoughts (response 2 & 3) etc. compared to the control group).
Related example: Jing, H. G., Szpunar, K. K., & Schacter, D. L.(2016). Interpolated Testing Influences Focused Attention and Improves Integration of Information During a Video-Recorded Lecture. Journal of Experimental Psychology: Applied, 22(3), 305–318. https://doi.org/10.1037/xap0000087 See: Experiment 2 Results: Thought probes and note taking.
on-task, task-related interference, or task-unrelated thoughtsthen ANOVA isn't correct as ANOVA assumes a continuous outcome. There are ways, however, to deal with categorical outcomes. Whether you need to do "repeated measures" analysis depends on how each individual is exposed to the "content-based thought probes" over time. Please explain that, too. $\endgroup$