ANOVA's and reaction time data I've been reading a lot on the effect of perceptual load on selective attention but the analysis confuses me somewhat. Some of these studies flag 20-40ms differences between conditions as significant where the mean is about 500ms and my gut tells me something is fishy. Unfortunately, almost no one reports any information about the sample, anything error, SD, CI etc. Most just report sample size (usually between 10-20) and mean RT. Looking at my semi-pilot data I'd be seriously hesitant to run ANOVA's on it. It's properly positively skewed (log and inverse transformations alleviate the situation somewhat, but not always), the residuals are pretty non-normal in some cases, the dispersion of the reaction times between certain participants differs quite significantly making me wonder if I'm not going to commit a whole host of type-1 errors when running repeated -measures ANOVA's. 
I can see why something like fixed effects models would be preferable in these circumstance, but almost no one does that. 
Here is a typical example:

"A 3x2 within-subjects ANOVA was conducted on mean correct RTs and
  error rates, with distractor condition (low dilution vs. high dilution
  vs. high dilution/high salience) and distractor compatibility
  (incompatible vs. neutral) as the within-subjects variables. All
  interpretations of the data were identical using both the Sphericity
  assumption and the Greenhouse-Geisser method. With respect to mean
  correct RTs, there was no main effect of distractor condition (F < 1),
  but there was a significant main effect. of distractor compatibility,
  F(1, 19)  10.06, p <.01, np2 = 0.35, indicating that RTs were
  faster with neutral distractors (506 ms) than incompatible distractors
  (517 ms). This main effect is qualified by a significant distractor
  condition by distractor compatibility interaction, F(2, 38) = 6.07, p
  <.01, np2 = 0.24." (Biggs and Gibson, 2014).

Is the central limit theorem really going to make it justifiable to run ANOVA's on data from 15 participants where the distributions are positively skewed and there are differences in the variability of the reaction times between participants? This between-participant variability obviously get's chucked out the window when you aggregate these scores to produce a mean for each participant per condition. 
I know I'm still a noob, but I'm having a hard time making sense of this so any advice/clarification would be greatly appreciated.  
 A: Most studies calculate mean or median for each subject and condition, then do statistics on the means. The central limit theorem comes to the rescue here. But if you are modelling individual trial RTs, e.g. in a trialwise hierarchical mixed model, then normality of residuals is essential.  
It is logically fine for reaction time differences of 5 ms to be significant even if 


*

*the mean RT is 500 ms, and 

*the between-subjects variability in mean RT is > 100 ms, and

*the within-subject trial-to-trial variability in RT is > 200 ms


all of which are unfortunately common in psychology.
Point 2 is saved by the use of repeated-measures designs, in which the variabiliy in mean RT across subjects is estimated and factored at a higher hierarchical level than the fixed effects. 
Point 3 is saved by the use of several 100s of trials per condition. Why RTs are so variable from trial to trial remains more-or-less unknown.RHS Carpenter, MLL Williams - Nature, 1995.
However I would be surprised if fewer than 20 subjects would give sufficient power to detect effects this small. This probably constitutes part of the replication crisis in these sciences. 
