I performed an experiment in which a group of $N=42$ participants performed a task under three different sound conditions. My data meets all the requirements for a RANOVA which confirmed that the performance, i.e. the number of errors in the task, significantly differs between the three sound conditions, Bonferroni adjusted post-hoc tests showed a significant difference for one pair of sound conditions.

At this point I could probably just end my analysis but I wondered if it makes sense to additionally evaluate under which condition each participant performed the best. This gives me counts for each sound condition, e.g. 23 participants performed best under the condition Silence (see Figure 1). Cases in which a participant achieved the lowest number of errors under multiple sound conditions were added to the count for each of those conditions. Assuming that the sound condition does not influence the results at all, I would expect these counts to be uniformly distributed between all three conditions? In this case, it should be possible to calculate the probability $P(X \leq k)$ to obtain $k$ or less counts $X$ for a single category using the binomial cumulative distribution function as

\begin{equation} P(X \leq k) = \sum_{i=0}^k \begin{pmatrix} n \\ i \end{pmatrix}p^i(1-p)^{n-i} \end{equation}

with $p=\frac{1}{3}$ and $n$ being equal to the sum of counts of all three conditions? Similarly one could calculate the probability to obtain $k$ or more counts for a single condition, Figure 1 shows the lowest of those two calculated values $P(X \leq k)$ and $P(X \geq k)$ for each condition.

Is it correct to assume that if this probability for the observed counts of a specific condition is small, e.g. $P < .05$, this sound condition eventually influenced the participants' performance or am I thinking a bit too simple here? If this is a valid approach, does it give me any meaningful insights in addition to my RANOVA results?

Thanks for your help!

Edit: I got some help here, in case someone else has a similar problem.

Fig. 1: Counts of participants that achieved lowest number of errors for each sound condition, p values calculated using binomial cdf.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy