Should I adjust p-values when investigating an ANOVA interaction?

I'm conducting an experiment to compare 4 types of headphones. I'm interested in finding out whether the headphones differ in their subjective quality.

Thirty subjects listened to each pair of headphones several times and rated the headphones' quality. My dependent variable is sound quality, and my two independent variables are headphone type (3 kinds) and subject (30 subjects). For each headphone-subject combination, I collected 12 ratings (12 replications).

A two-way anova produced the following table. It shows a significant interaction effect between headphone and subject. This seems to indicate that the subjects didn't agree on their ratings of the headphones.

                        Df Sum Sq Mean Sq F value   Pr(>F)
headphone                2   2933    1466   4.116   0.0165 *
subject                 29 165500    5707  16.022  < 2e-16 ***
headphone:subject       58  46897     809   2.270 3.24e-07 ***
Residuals             1350 480869     356


An examination of the data showed that many subjects failed to differentiate between the headphones. That is, the quality ratings they gave to each didn't seem different from one another. The graphic below, for example, shows data from five subjects. The medians (red lines) seem roughly equal for subjects 1 and 5.

The next question I'd like to answer is "how many of the 30 subjects could reliably discriminate between the 3 headphone types". Or, in more statistical terms, "for how many subjects can I reject the null hypothesis that mean ratings are equal".

My first instinct for answering this question was to run a one-way anova on each subject (IV, headphone type), and reject the null if $p$ was below my $\alpha$ (0.05). On further reflection, I realized that this approach would give me many false-positives: some subjects would be likely to yield $p < 0.05$ by chance alone.

If I want to determine how many subjects could discriminate between the headphones, should I use adjusted p-values for these 30 anovas? (I.e. should I adjust the $p$s to address the the multiple comparisons problem using, for example, Holm-Bonferroni correction?)

3 Answers

The next question I'd like to answer is "how many of the 30 subjects could reliably discriminate between the 3 headphone types"

Yes, using $p<0.05$ criterion will lead to several false positives expected by chance alone. You should either use some formal method of multiple testing adjustments, or perhaps simply lower the cutoff to some more conservative but still conventional value, such as e.g. $p<0.01$ or even $p<0.001$.

In addition or even instead, I would suggest to look at the $p$-values for all 30 subjects. Then instead of writing that e.g. "23 out of 30 subjects could reliably discriminate between headphone types" you will be able to say something like "20 subjects could clearly discriminate between headphone types ($p<0.001$), 7 subjects clearly could not ($p>0.1$) and 3 subjects fell somewhere in between".

Finally, note that even a very small $p$-value does not mean that the discrimination is "reliable". To me, "reliable" refers rather to the effect size, e.g. I would call say that a subject who can name the headphone type with above e.g. 90% accuracy is reliable. But with 12 repetitions for each type, you can get significant (even highly significant) but still very small difference, corresponding e.g. to the 40% accuracy. It can well be above chance (33%), but is hardly reliable.

This may not fully answer your question, but you might look into mixed-effects models, which allow you to control for subject-level variation ("Joe always estimates high") without actually estimating coefficients for each subject. This lets you get at the original question you posed: "Do the headphones differ in subjective quality?" without running thirty ANOVAs and trying to compare the results.

Normally subjects are kept in the Error term to reduce the effect of subject variability but your aim is to determine which subjects are significantly different from others. You can take help of graphics to identify such subjects. For each subject you can determine range of rating given (max rating - min rating) and plot them:

Then you can determine if there are any outliers or those who lie below 2.5 or above 97.5th percentiles.

Also, plots from regression analysis can be used. If the data is arranged as follows (note data is different from that in above plot):

   subject variable value
1        A  headph1     4
2        B  headph1     5
3        C  headph1     6
4        D  headph1     5
5        E  headph1     4
6        F  headph1     2

mod = lm(value~variable+subject, mydata)


Residual plots will show which readings are outliers:

plot(mod)


Note that outliers are numbered on the plots above.

> library(car)
> crPlots(mod)


Note subject F is clearly standing out here.