I'm conducting an experiment to compare 4 types of headphones. I'm interested in finding out whether the headphones differ in their subjective quality.
Thirty subjects listened to each pair of headphones several times and rated the headphones' quality. My dependent variable is sound quality, and my two independent variables are headphone type (3 kinds) and subject (30 subjects). For each headphone-subject combination, I collected 12 ratings (12 replications).
A two-way anova produced the following table. It shows a significant interaction effect between headphone and subject. This seems to indicate that the subjects didn't agree on their ratings of the headphones.
Df Sum Sq Mean Sq F value Pr(>F) headphone 2 2933 1466 4.116 0.0165 * subject 29 165500 5707 16.022 < 2e-16 *** headphone:subject 58 46897 809 2.270 3.24e-07 *** Residuals 1350 480869 356
An examination of the data showed that many subjects failed to differentiate between the headphones. That is, the quality ratings they gave to each didn't seem different from one another. The graphic below, for example, shows data from five subjects. The medians (red lines) seem roughly equal for subjects 1 and 5.
The next question I'd like to answer is "how many of the 30 subjects could reliably discriminate between the 3 headphone types". Or, in more statistical terms, "for how many subjects can I reject the null hypothesis that mean ratings are equal".
My first instinct for answering this question was to run a one-way anova on each subject (IV, headphone type), and reject the null if $p$ was below my $\alpha$ (0.05). On further reflection, I realized that this approach would give me many false-positives: some subjects would be likely to yield $p < 0.05$ by chance alone.
If I want to determine how many subjects could discriminate between the headphones, should I use adjusted p-values for these 30 anovas? (I.e. should I adjust the $p$s to address the the multiple comparisons problem using, for example, Holm-Bonferroni correction?)