I recently ran a perceptual study in which I aimed to measure how emotion recognition accuracy from whole-body motions changed when two type of body representations were used. Five actors Ai with i=1,..,5 performed 5 motion sequences Mj with j=1,..,5 for 5 emotional states Ek with k=1,..,5; a total of 125 needed to be evaluated for each body representation. Since the motion sequences were too long, I could not ask a rater to watch all 250 videos at once. Instead, I randomly defined 5 groups such that one rater would only see 25 videos in total, 5 for each emotion. I followed a design similar to the one presented here.
The same groups were used for both body representations and an rater would only see one body representation. I have a total of 20 ratings for each group. The groups look something like this:
Actor | Sequence
M1 M2 M3 M4 M5 A1 1 1 2 2 2 A2 1 1 1 2 2 A3 3 3 3 4 4 A4 3 3 4 4 4 A5 5 5 5 5 5
Since the reason to divide the motion sequences and actors into groups was to reduce raters' boredom and fatigue, my intuition tells me that I can easily combine the results for each body representation and use an ANOVA (repeated measures) analysis with representation as between-subject factor and emotion as within-subject factor. However, I would also like to see if there is any effect due to actors and/or motion sequences. Can I combine answers according to actors and/or sequences and apply the same ANOVA analysis with actor or sequence as a between-subject factor? Or should I use a more complex analysis method, for example to consider actor and motion as random effects? I'm not sure about the ANOVA because at the end some raters saw more than one actor or sequence.
I found a question similar to mine Data analysis for mixed within-subjects/between-subjects design?. However, the sequences were not chosen uniformly at random and independently for each rater, so I am not sure I can apply the proposed solution.