3
$\begingroup$

Assume a hypothetical scenario of two events. During event 1, I observe a set of values $[X_1, X_2,X_3,...,X_n]$. A physical phenomena occurs and this triggers event 2 for a short period of time and I end up observing another set of values $[Y_1, Y_2,Y_3,...,Y_n]$ during that time. I don't know anything about the underlying distributions.

I have a number of $X-Y$ pairs and I want a method to detect when the Y values are generally larger than the $X$ values for further investigation. The pairs are not comparable - that is I cannot use the $X$ values from another subject to inform me about the distribution, however, in the null case I expect $X$ and $Y$ should be from the approximately the same distribution.

My first attempt is to define event 2 to have some importance if the median of the observed values from event 2 is higher than that of the observed values in event 1.

I have a few questions:

  • Because median is known to be robust to outliers, what are the pitfalls of such a comparison i.e., when will this fail?
  • If this is not good, then can I make use of something like KL-Divergence or JS-Divergence to understand the extent of divergence (and if it is beyond a threshold, conclude that event 2 is important)?
  • Does comparing divergence have any advantages over the median?

I think I want to detect significant change as an event. Any clarification will be greatly appreciated.

$\endgroup$
4
  • 3
    $\begingroup$ The most obvious drawback of your approach is that it has a 50% false positive rate. $\endgroup$
    – whuber
    Jan 18, 2013 at 21:30
  • $\begingroup$ @whuber: +1 Thank you for your comment. Pardon me for my ignorance but can you please elaborate on this? Is there a reference I can read that goes in-depth? I was thinking median is robust to outliers. $\endgroup$
    – Legend
    Jan 18, 2013 at 21:38
  • 1
    $\begingroup$ Your question is a good one but it's extremely broad: you are really asking about how to conduct a non-parametric hypothesis test. $\endgroup$
    – whuber
    Jan 18, 2013 at 21:44
  • $\begingroup$ @whuber: Also, if you have some time, can you please clarify/elaborate on your previous statement: The most obvious drawback of your approach is that it has a 50% false positive rate? Even a reference might suffice. Thank you. $\endgroup$
    – Legend
    Jan 19, 2013 at 7:07

1 Answer 1

5
+50
$\begingroup$

As pointed out by @whuber, as currently described your test would have a 50% false positive rate, which would usually unacceptable. This is simply because, suppose X and Y are both drawn from the same distribution, then just by sample noise the will almost surely have different medians. 50% of the time X will have the higher sample median and 50% of the time Y will, therefore 50% of the time you will (falsely) reject the hypothesis that they are the same.

Of course, this is easily fixed by considering if the median is significantly higher for the second set than the first. One simple way of doing this would be to resample (bootstrap perhaps) the X distribution to find the median distribution. Then simply reject when the median is over the 95th percentile. This technique could readily be adapted to any test statistic that you decide to be interested in.

For example:

> X <- c(10.1,12.2,14.3,13.4,17.5,21.6,14.7, 9.8, 8.9, 7.0)
> Y <- c(11.1,22.2,18.3,15.4,13.5,17.6,20.7,21.8,14.9, 8.0)
> MedianSamples <- replicate(2000,median(sample(X, replace = TRUE)))
> summary(MedianSamples)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   7.95   11.15   12.80   12.56   13.85   21.60 
> Y.median <- median(Y)
> mean(MedianSamples > Y.median)
[1] 0.0135

The problem here, however, is that the sample size is so small that the bootstrap resample is probably not very good.

If you believe that each $X_i$ is drawn from the same distribution within an event KLD or JSD would indeed be a good ways to measure difference in distribution. JSD might be particularly natural here, since JSD will tell you the mutual information between the indicator (X or Y) and the mixed distribution.

However since this is time-series data, and specifically, non-stationary data, then those assumptions do not hold. In that case, then KLD and JSD would not be suitible, since they assume that each $X_i$ in $[X_1...X_n]$ is drawn from the same distribution.

I think the best tool for this job would be a Wilcoxon Rank test. This is a non-parametric test of whether one sample is "generally larger" than the other sample, without making any distributional assumptions. It also generally behaves fine for small samples. This is implemented in R: http://stat.ethz.ch/R-manual/R-patched/library/stats/html/wilcox.test.html

> wilcox.test(Y, X, paired = FALSE, alternative = "greater")

    Wilcoxon rank sum test

data:  Y and X 
W = 74, p-value = 0.03763
alternative hypothesis: true location shift is greater than 0 
$\endgroup$
11
  • $\begingroup$ +1 This is an excellent answer. I appreciate your suggestion about using bootstrapping to test for median significance. Do you know if this can be done out-of-the-box in R? I mean, are there any standard functions to do this? To answer your question, yes, I am interested in the median of X being higher than that of Y. As per your last comment: This data is the amount of water flowing on a pipe taken on an hourly basis. Considering that stationarity is not seen in real-world processes, do you have any suggestions on how to handle this? $\endgroup$
    – Legend
    Jan 28, 2013 at 21:09
  • $\begingroup$ Also, to be fair, I have already awarded the reputation bounty for your answer as it was of great help. I'll accept this as an answer shortly after getting some more details. In regard to your reply, can you also comment anything on false negatives or is this not possible? $\endgroup$
    – Legend
    Jan 28, 2013 at 21:32
  • $\begingroup$ Ok, thanks. When I get a moment I'll try edit my answer to make it better match your problem. In short, yes there are many ways to do bootstrap in R. The sample command will probably suffice in your case; I will try to give you an example. There is also the boot package but that will be overkill for your needs. $\endgroup$
    – Corvus
    Jan 28, 2013 at 21:42
  • $\begingroup$ As to the non-stationarity and median issue: could you explain a little more about the data? The $X_i$ are sequential hourly flows from one pipe, and the $Y_i$ are also sequential hourly flows from one pipe? Is it the same pipe? Is it two different days? Do you do this many times (i.e. may X vectors and many Y vectors, or 1 "reference" X and many Ys, or just a one off: 1 X 1 Y)? $\endgroup$
    – Corvus
    Jan 28, 2013 at 21:45
  • $\begingroup$ Great thank you! The data is the amount of water flowing on one pipe on an hourly basis and I have many pipes. Intuitively, to know if something "surprising" happened, I was thinking it would make sense to compare the distributions rather than the medians (because many distributions can also have the same median). To achieve this, I thought of comparing the distribution of water flows in the last 10 hours with the water flows recorded during the event. $\endgroup$
    – Legend
    Jan 28, 2013 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.