How to compare distributions for observed responses?

Question

I performed an experiment in which 37 participants were asked 24 questions and their eye movements were recorded while they solved each question. For each question, I created distributions of a particular eye movement (call it x) over time for participants who answered right and those who answered wrong. I divided each participant's time range into 10 bins (representing 0-10%, 0-20%, 0-30% of time until 0-100% of time). For example, if 10 participants answered a question correctly and 1 out of them had x movement in the first 10% of their time, 2 had x movement within the first 20% of their time, 5 had in the first 30% of their time and so on.., I represented the distribution as (1/10,2/10,5/10....), to get a sort of cumulative distribution function (the last value would be the proportion of participants who have had movement x at all during the course of their time).

For each question, I have two such distribution curves, one for participants who have answered correctly and the other for wrong participants. I want to see whether there is a significant difference between these two distributions. I have looked at multiple regression models and growth curve analysis models. Would they work here? What kind of techniques should I look at?

Answer 1

Your sample size is quite small, so it will be hard to compare CDFs as all tests I know assume $n \rightarrow \infty$ (like one below). In your case I would try doing ANOVA on some statistics. However, if you really want, you can try this idea (but you have to read about its behaviour in low $n$ setting):

You can compare two cumulative distribution functions using the Kolmongorov-Smirnov Statistic and performing the two sample Kolmongrov-Smirnov test.

This statistic shows the maximum difference between the two empirical CDFs:

\begin{equation} K = sup_{z \in \mathbb{R}} |\hat{F}_1(z) - \hat{F}_2(z)|. \end{equation}

for $n \rightarrow \infty$ the CDF of $\sqrt{n} K$ converges to the CDF

\begin{equation} R(k) = 1 - \sum_{i = 1}^{+ \infty} (-1)^{i-1} exp(-2i^2k^2). \end{equation}

I use it in MCMC where n is much larger than in your case.