Comparing distributions as opposed to medians?

Question

Assume a hypothetical scenario of two events. During event 1, I observe a set of values $[X_1, X_2,X_3,...,X_n]$. A physical phenomena occurs and this triggers event 2 for a short period of time and I end up observing another set of values $[Y_1, Y_2,Y_3,...,Y_n]$ during that time. I don't know anything about the underlying distributions.

I have a number of $X-Y$ pairs and I want a method to detect when the Y values are generally larger than the $X$ values for further investigation. The pairs are not comparable - that is I cannot use the $X$ values from another subject to inform me about the distribution, however, in the null case I expect $X$ and $Y$ should be from the approximately the same distribution.

My first attempt is to define event 2 to have some importance if the median of the observed values from event 2 is higher than that of the observed values in event 1.

I have a few questions:

• Because median is known to be robust to outliers, what are the pitfalls of such a comparison i.e., when will this fail?
• If this is not good, then can I make use of something like KL-Divergence or JS-Divergence to understand the extent of divergence (and if it is beyond a threshold, conclude that event 2 is important)?
• Does comparing divergence have any advantages over the median?

I think I want to detect significant change as an event. Any clarification will be greatly appreciated.

Answer 1

As pointed out by @whuber, as currently described your test would have a 50% false positive rate, which would usually unacceptable. This is simply because, suppose X and Y are both drawn from the same distribution, then just by sample noise the will almost surely have different medians. 50% of the time X will have the higher sample median and 50% of the time Y will, therefore 50% of the time you will (falsely) reject the hypothesis that they are the same.

Of course, this is easily fixed by considering if the median is significantly higher for the second set than the first. One simple way of doing this would be to resample (bootstrap perhaps) the X distribution to find the median distribution. Then simply reject when the median is over the 95th percentile. This technique could readily be adapted to any test statistic that you decide to be interested in.

For example:

> X <- c(10.1,12.2,14.3,13.4,17.5,21.6,14.7, 9.8, 8.9, 7.0)
> Y <- c(11.1,22.2,18.3,15.4,13.5,17.6,20.7,21.8,14.9, 8.0)
> MedianSamples <- replicate(2000,median(sample(X, replace = TRUE)))
> summary(MedianSamples)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   7.95   11.15   12.80   12.56   13.85   21.60 
> Y.median <- median(Y)
> mean(MedianSamples > Y.median)
[1] 0.0135

The problem here, however, is that the sample size is so small that the bootstrap resample is probably not very good.

If you believe that each $X_i$ is drawn from the same distribution within an event KLD or JSD would indeed be a good ways to measure difference in distribution. JSD might be particularly natural here, since JSD will tell you the mutual information between the indicator (X or Y) and the mixed distribution.

However since this is time-series data, and specifically, non-stationary data, then those assumptions do not hold. In that case, then KLD and JSD would not be suitible, since they assume that each $X_i$ in $[X_1...X_n]$ is drawn from the same distribution.

I think the best tool for this job would be a Wilcoxon Rank test. This is a non-parametric test of whether one sample is "generally larger" than the other sample, without making any distributional assumptions. It also generally behaves fine for small samples. This is implemented in R: http://stat.ethz.ch/R-manual/R-patched/library/stats/html/wilcox.test.html

> wilcox.test(Y, X, paired = FALSE, alternative = "greater")

    Wilcoxon rank sum test

data:  Y and X 
W = 74, p-value = 0.03763
alternative hypothesis: true location shift is greater than 0