The title might be a bit misleading. Unfortunately statistics is not my area of forte, so gentle guidance along the right path is much appreciated. That said, here's my problem:
I'm working on analyzing precise time measurements from an experimental process. I have 16 measurement groups. Of these, one should (on average) differ slightly in magnitude from the others. The remaining 15 should have similar or identical properties. I'm trying to determine which of these measurement groups is the odd one out. My first pass at the problem involved a naive data collection routine rotating between sample groups until enough samples from each group were collected.
My data is clearly not normal. Furthermore, sample values seem to fluctuate as a function of time (non-scientific visual inspection shows that larger values and smaller values tend to cluster across sample groups). The variable under consideration should not be fluctuating like this, so I believe this effect is attributable to some other element in an admittedly noisy environment.
My first pass at analysis was to run a KS test comparing each sample group to the set consisting of all other collected data. This worked as expected: the sample which was known to vary showed a much lower p-value than the other candidates.
The KS test isn't ideal here. Despite a theoretical belief that the data is continuous, my measurement resolution limits dictate that there must be a number of duplicate data points (3 effective places of resolution and n between 10k and 1m per group).
Unfortunately, this sample set was taken from a somewhat idealized experimental condition. I need to improve my analysis to include some larger elements of noise. I have done the experiment in my real-world condition, and have been having trouble finding reliable ways to analyze my data.
The problem seems to be related to the fact that while my effect remains constant over much larger (and thus longer-running) sampling regimes, the macro-level time changes seem to be interfering. As I move to an order of magnitude more samples in my real-world condition, rather than converging on one single value as different, the KS test starts to show that ALL the values differ (and the ordering is inconsistent).
I've re-written my data collection to log both the timestamp for each sample, and several timing values taken over the course of my experimental procedure (originally I was just logging a single aggregate timing result by calculating a difference using these values). With this expanded data in hand, I've been looking for appropriate analysis techniques.
Here's where I'm stumped. I'm not sure what I should do with this timestamp data. I've turned a multiple sample comparison problem into a time sequence. If I can use the time data to even out my values, I'm still (at least potentially) left with a complicated multi-variable analysis. PCA assumes normality so that's out. ICA isn't appropriate because I know that most of my measured variables are not, in fact, independent of each other. Once I've sorted that out, I still need to compare to the overall group and figure out which one is different. Because each sample-per-group has a discrete timestamp, I don't have a simple way to compare across my sampling groups.
It seems like the Wilcoxon–Mann–Whitney (Wilcoxon rank-sum) test is likely to be of use here, but only after I've done work on the underlying data. Suggestions?
Edit: for what it's worth, this is a practical problem. Approximate solutions and things that narrow the candidate field are perfectly reasonable. No points for excessively pretty P values here.
