1
$\begingroup$

The title might be a bit misleading. Unfortunately statistics is not my area of forte, so gentle guidance along the right path is much appreciated. That said, here's my problem:

I'm working on analyzing precise time measurements from an experimental process. I have 16 measurement groups. Of these, one should (on average) differ slightly in magnitude from the others. The remaining 15 should have similar or identical properties. I'm trying to determine which of these measurement groups is the odd one out. My first pass at the problem involved a naive data collection routine rotating between sample groups until enough samples from each group were collected.

My data is clearly not normal. Furthermore, sample values seem to fluctuate as a function of time (non-scientific visual inspection shows that larger values and smaller values tend to cluster across sample groups). The variable under consideration should not be fluctuating like this, so I believe this effect is attributable to some other element in an admittedly noisy environment.

My first pass at analysis was to run a KS test comparing each sample group to the set consisting of all other collected data. This worked as expected: the sample which was known to vary showed a much lower p-value than the other candidates.

The KS test isn't ideal here. Despite a theoretical belief that the data is continuous, my measurement resolution limits dictate that there must be a number of duplicate data points (3 effective places of resolution and n between 10k and 1m per group).

Unfortunately, this sample set was taken from a somewhat idealized experimental condition. I need to improve my analysis to include some larger elements of noise. I have done the experiment in my real-world condition, and have been having trouble finding reliable ways to analyze my data.

The problem seems to be related to the fact that while my effect remains constant over much larger (and thus longer-running) sampling regimes, the macro-level time changes seem to be interfering. As I move to an order of magnitude more samples in my real-world condition, rather than converging on one single value as different, the KS test starts to show that ALL the values differ (and the ordering is inconsistent).

I've re-written my data collection to log both the timestamp for each sample, and several timing values taken over the course of my experimental procedure (originally I was just logging a single aggregate timing result by calculating a difference using these values). With this expanded data in hand, I've been looking for appropriate analysis techniques.

Here's where I'm stumped. I'm not sure what I should do with this timestamp data. I've turned a multiple sample comparison problem into a time sequence. If I can use the time data to even out my values, I'm still (at least potentially) left with a complicated multi-variable analysis. PCA assumes normality so that's out. ICA isn't appropriate because I know that most of my measured variables are not, in fact, independent of each other. Once I've sorted that out, I still need to compare to the overall group and figure out which one is different. Because each sample-per-group has a discrete timestamp, I don't have a simple way to compare across my sampling groups.

It seems like the Wilcoxon–Mann–Whitney (Wilcoxon rank-sum) test is likely to be of use here, but only after I've done work on the underlying data. Suggestions?

Edit: for what it's worth, this is a practical problem. Approximate solutions and things that narrow the candidate field are perfectly reasonable. No points for excessively pretty P values here.

$\endgroup$
  • $\begingroup$ What do you want to know? I don't see a clear question here. $\endgroup$ – whuber Dec 14 '10 at 18:21
  • $\begingroup$ I'm trying to figure out which of my sample groups displays different statistical properties (in this case by taking slightly longer) from the rest. There is exactly one group for which this is known to be the case, all other groups come from the same distribution. $\endgroup$ – Paul McMillan Dec 14 '10 at 18:28
  • $\begingroup$ @Paul "Statistical properties" in what sense? What role does the timing play: is it an auxiliary variable or is it the variable of interest? I would find it much easier to think about your question if you were to reformulate it without the extensive speculation about types of tests and simply describe as clearly as possible what your experiment consists of and what you want to learn from it. $\endgroup$ – whuber Dec 14 '10 at 18:57
  • $\begingroup$ I have a process which takes a certain amount of time to run depending on the input value. I have 16 potential inputs. Of these, one will make the process run for a slightly longer period of time than the others. I need to identify this input. I am attempting to do so by running many tests, and recording the time my process takes for each test. $\endgroup$ – Paul McMillan Dec 14 '10 at 19:44
  • $\begingroup$ Unfortunately, my data is noisy. This noise manifests itself (in one of many ways) by periodically causing all samples (regardless of input) to take longer to process. Since I am sampling in a round-robin fashion, it seems likely that I may be able to correct for this. $\endgroup$ – Paul McMillan Dec 14 '10 at 19:49
3
$\begingroup$

From what I can tell, your problems seem to be:

1) smooth the time series data to remove correlated fluctuations.

2) Identify which of the inputs differs, using the smoothed data.

You're seem to be worried about not being able to solve (2) once you solve (1). But let's solve 1 first and then worry about 2, right?

Here's one idea. You say you're sampling in a round robin fashion and you have 16 inputs. So maybe treat each 16 draws as one "round", sum up all of the values in each round, and divide each value in the 16-draw round by that sum to normalize.

It seems that this would work if your time series data is correlated on a longer time scale than 16 data points. If the data is correlated on a much longer time scale you could even normalize in larger chunks, like 160 or 1,600 data points, to maximize noise reduction (and comp. efficiency).

$\endgroup$
  • $\begingroup$ Good point - smoothing the data is the first problem, since I already have some methods that work for the second part. I'll give that a go and see how it does. $\endgroup$ – Paul McMillan Dec 17 '10 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.