Assume a hypothetical scenario of two events. During event 1, I observe a set of values $[X_1, X_2,X_3,...,X_n]$. A physical phenomena occurs and this triggers event 2 for a short period of time and I end up observing another set of values $[Y_1, Y_2,Y_3,...,Y_n]$ during that time. I don't know anything about the underlying distributions.
I have a number of $X-Y$ pairs and I want a method to detect when the Y values are generally larger than the $X$ values for further investigation. The pairs are not comparable - that is I cannot use the $X$ values from another subject to inform me about the distribution, however, in the null case I expect $X$ and $Y$ should be from the approximately the same distribution.
My first attempt is to define event 2 to have some importance if the median of the observed values from event 2 is higher than that of the observed values in event 1.
I have a few questions:
- Because median is known to be robust to outliers, what are the pitfalls of such a comparison i.e., when will this fail?
- If this is not good, then can I make use of something like KL-Divergence or JS-Divergence to understand the extent of divergence (and if it is beyond a threshold, conclude that event 2 is important)?
- Does comparing divergence have any advantages over the median?
I think I want to detect significant change as an event. Any clarification will be greatly appreciated.