I've been reading a lot on the effect of perceptual load on selective attention but the analysis confuses me somewhat. Some of these studies flag 20-40ms differences between conditions as significant where the mean is about 500ms and my gut tells me something is fishy. Unfortunately, almost no one reports any information about the sample, anything error, SD, CI etc. Most just report sample size (usually between 10-20) and mean RT. Looking at my semi-pilot data I'd be seriously hesitant to run ANOVA's on it. It's properly positively skewed (log and inverse transformations alleviate the situation somewhat, but not always), the residuals are pretty non-normal in some cases, the dispersion of the reaction times between certain participants differs quite significantly making me wonder if I'm not going to commit a whole host of type-1 errors when running repeated -measures ANOVA's.
I can see why something like fixed effects models would be preferable in these circumstance, but almost no one does that.
Here is a typical example:
"A 3x2 within-subjects ANOVA was conducted on mean correct RTs and error rates, with distractor condition (low dilution vs. high dilution vs. high dilution/high salience) and distractor compatibility (incompatible vs. neutral) as the within-subjects variables. All interpretations of the data were identical using both the Sphericity assumption and the Greenhouse-Geisser method. With respect to mean correct RTs, there was no main effect of distractor condition (F < 1), but there was a significant main effect. of distractor compatibility, F(1, 19) 10.06, p <.01, np2 = 0.35, indicating that RTs were faster with neutral distractors (506 ms) than incompatible distractors (517 ms). This main effect is qualified by a significant distractor condition by distractor compatibility interaction, F(2, 38) = 6.07, p <.01, np2 = 0.24." (Biggs and Gibson, 2014).
Is the central limit theorem really going to make it justifiable to run ANOVA's on data from 15 participants where the distributions are positively skewed and there are differences in the variability of the reaction times between participants? This between-participant variability obviously get's chucked out the window when you aggregate these scores to produce a mean for each participant per condition.
I know I'm still a noob, but I'm having a hard time making sense of this so any advice/clarification would be greatly appreciated.