2
$\begingroup$

I've been reading a lot on the effect of perceptual load on selective attention but the analysis confuses me somewhat. Some of these studies flag 20-40ms differences between conditions as significant where the mean is about 500ms and my gut tells me something is fishy. Unfortunately, almost no one reports any information about the sample, anything error, SD, CI etc. Most just report sample size (usually between 10-20) and mean RT. Looking at my semi-pilot data I'd be seriously hesitant to run ANOVA's on it. It's properly positively skewed (log and inverse transformations alleviate the situation somewhat, but not always), the residuals are pretty non-normal in some cases, the dispersion of the reaction times between certain participants differs quite significantly making me wonder if I'm not going to commit a whole host of type-1 errors when running repeated -measures ANOVA's.

I can see why something like fixed effects models would be preferable in these circumstance, but almost no one does that.

Here is a typical example:

"A 3x2 within-subjects ANOVA was conducted on mean correct RTs and error rates, with distractor condition (low dilution vs. high dilution vs. high dilution/high salience) and distractor compatibility (incompatible vs. neutral) as the within-subjects variables. All interpretations of the data were identical using both the Sphericity assumption and the Greenhouse-Geisser method. With respect to mean correct RTs, there was no main effect of distractor condition (F < 1), but there was a significant main effect. of distractor compatibility, F(1, 19) 10.06, p <.01, np2 = 0.35, indicating that RTs were faster with neutral distractors (506 ms) than incompatible distractors (517 ms). This main effect is qualified by a significant distractor condition by distractor compatibility interaction, F(2, 38) = 6.07, p <.01, np2 = 0.24." (Biggs and Gibson, 2014).

Is the central limit theorem really going to make it justifiable to run ANOVA's on data from 15 participants where the distributions are positively skewed and there are differences in the variability of the reaction times between participants? This between-participant variability obviously get's chucked out the window when you aggregate these scores to produce a mean for each participant per condition.

I know I'm still a noob, but I'm having a hard time making sense of this so any advice/clarification would be greatly appreciated.

$\endgroup$
  • 2
    $\begingroup$ Inverse reaction times (speeds) may be a better target, but either way I'd be inclined to consider GLMs for times or speeds. With the individual variability, you might even do better on a log scale, but then I'd be looking at a mixed model (perhaps random intercepts in the logs?). More specific advice is difficult without data. $\endgroup$ – Glen_b -Reinstate Monica Aug 19 '14 at 5:34
  • 1
    $\begingroup$ @Glen_b Thanks for the advice. I'm actually thinking of going the fixed effects route and allow for random intercepts for participants. I'm just surprised that almost everyone keeps running repeated-measures ANOVA's on what are potentially problematic data and don't report any information apart from some semi-cryptic error bars in bar graphs. $\endgroup$ – Rooirokbokkie Aug 19 '14 at 5:53
  • 1
    $\begingroup$ Perhaps you could contact the authors and get more complete information on the statistical analyses they report. You might also be able to get the raw data. $\endgroup$ – Joel W. Aug 20 '14 at 13:22
1
$\begingroup$

Most studies calculate mean or median for each subject and condition, then do statistics on the means. The central limit theorem comes to the rescue here. But if you are modelling individual trial RTs, e.g. in a trialwise hierarchical mixed model, then normality of residuals is essential.

It is logically fine for reaction time differences of 5 ms to be significant even if

  1. the mean RT is 500 ms, and
  2. the between-subjects variability in mean RT is > 100 ms, and
  3. the within-subject trial-to-trial variability in RT is > 200 ms

all of which are unfortunately common in psychology.

Point 2 is saved by the use of repeated-measures designs, in which the variabiliy in mean RT across subjects is estimated and factored at a higher hierarchical level than the fixed effects.

Point 3 is saved by the use of several 100s of trials per condition. Why RTs are so variable from trial to trial remains more-or-less unknown.RHS Carpenter, MLL Williams - Nature, 1995.

However I would be surprised if fewer than 20 subjects would give sufficient power to detect effects this small. This probably constitutes part of the replication crisis in these sciences.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.