4
$\begingroup$

I am have a collection of events recorded in two separate conditions, and I have measured the duration of each one of them. I would like to know two things: 1) are the distributions of durations different between the two conditions, and 2) if so, how are they different? For example, maybe the biggest difference between the two conditions is that there are far more 10 ms long events in condition #1 than in condition #2.

I feel confident in using a Kolmogorov-Smirnov test to determine whether the distributions are significantly different, but am stuck on how to describe what the differences actually are using a statistical test or collection of test statistics. I could histogram the data and run some sort of ANOVA, but that seems a little course and inelegant. Or maybe it is fair to plot the distributions on top of each other and simply read off the difference at areas of interest. Any advice on how to tackle this problem?

Below is some example data. Here a Kolmogorov-Smirnov test comparing these two distributions returns a very small p-value (p < 0.000001), so I know that the two distributions come from different populations. Based on how I think the KS test works, this also tells me that where the two distributions are widest apart in the horizontal plane (around 0.9 on the y-axis) the two distributions are significantly different (there are more events with durations around 60 ms in group 1 than in group 2). But what about everywhere else in the distribution? Is there a way for me to know whether the difference in the amount of events that are 10 ms long is also statistically significant?

enter image description here

Improve this question
$\endgroup$
7
  • $\begingroup$ There are many ways in which distributions might differ. It depends on the kinds of differences that are important to describe. $\endgroup$ – Glen_b Jun 13 '14 at 4:41
  • $\begingroup$ It might be useful to make a QQ-plot of the two distributions! $\endgroup$ – kjetil b halvorsen♦ Jun 13 '14 at 8:21
  • $\begingroup$ You could look into " "Relative Distribution Methods in the Social Sciences" by Mark S. Handcock and Martina Morris, Springer-Verlag, 1999" which has an associated R package, reldist. $\endgroup$ – kjetil b halvorsen♦ Jun 13 '14 at 8:23
  • $\begingroup$ @ kjetil b halvorsen: Thanks for the suggestions! I've looked into Relative Distributions and QQ-plots, but unless I'm misunderstanding them, I don't think they help me to answer the problem I'm after. In both cases I'm visually interpreting a plot, but don't have a way of testing the difference between the distributions at every position along the curve? Is that correct? $\endgroup$ – stat_quest Jun 27 '14 at 18:28
  • $\begingroup$ I've edited the question to include some sample data and potentially clarifying remarks. $\endgroup$ – stat_quest Jun 27 '14 at 18:30
4
$\begingroup$

From a descriptive standpoint, a QQ plot is about as good as it gets for something like this, you can really see where in the distributions things are different (heavier tails, skew, location shift, outliers). That might give you insight about what parameter of the distributions you are interested in testing.

The issue with a statistical test is that it demands that you really clearly state what you care about, i.e., your null hypothesis. As an example, if you wanted to compare whether or not the mean of two distributions was different, you might not care that their standard deviations are different: 

a = rnorm(100, sd = 3); b = rnorm(100, sd = 1))
> t.test(a,b)$p.val
[1] 0.2825168

> ks.test(a,b)$p.val
[1] 7.14257e-05

If our null hypothesis is that the means are different, it wouldn't be correct to test with the KS-test and we see we'd incorrectly reject the null. In your case, you have to state, what aspect of the distribution you care about and then test accordingly.

It would be fine to ask, Is the 10th-percentile different -- that's a good null hypothesis. I'm not aware of well-known statistical test for such a thing, but you could certainly use a bootstrap-based approach.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Is it fair to do a KS-test first, then follow up with testing differences between specific areas of the distributions? I'm envisioning the protocol might be similar to the way one does an ANOVA first, then follows up with multiple comparisons. $\endgroup$ – stat_quest Jun 30 '14 at 3:17
  • $\begingroup$ Can you expand on the "bootstrap-based approach" suggestion, please? $\endgroup$ – stat_quest Jun 30 '14 at 3:19
  • $\begingroup$ Sure, I might not have enough characters, but: Your null is that the two distributions have the same 10th percentile. You compute the 10th percentile on each of your two distributions take the absolute difference, call that absolute difference S*. You now sample, with replacement, two datasets of equal size as your originals from the combined dataset (under the null, they have the same distribution), compute the abs 10th percentile difference as before; S_1,..._S_100. Count how many of the times your original S is bigger than the 100 resampled differences (S_1, ..., S_100). $\endgroup$ – jimmyb Jul 1 '14 at 0:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.