I have a problem that people here might be able to help me solve. I have come across a variety of answers on CrossValidated and StackOverflow, but none that quite match my current issue. Let me explain what I am trying to compare, and our current method.
I have a set of experimental length versus time traces (L(t) vs. t) from numerous experiments, all showing similar, but not identical, behavior. I then have a simulation model that is supposed to reproduce what is seen experimentally, however, there are numerous free parameters in this model. At each set of model parameters, I run several 'experiments' (this is a stochastic type computational simulation), and get L(t) traces too. What I want to know is how to compare these simulated traces to their experimental counterparts. Ideally, I would be able to do this in python, as that is where all of my other analysis code lives, but I can always program up something new. I'm not looking for trends in each individual time series, or if one can predict the other, but how well the simulated traces match what is seen experimentally.
Current Method I build a probability distribution of the L(t) traces for both experiment and simulations, and then use the 2-sample Kolmogrov-Smirnov test to get a p-value describing how different the two length distributions are. One of the main problems with this approach is that we lose some of the short time behavior of the length traces (i.e. the L(t) for small t doesn't match well between experiment and simulations).
Other Approaches I've found several other approaches that might work for me, but I was curious if people had ideas on other ways to approach this problem.
- Bin the time series into segments, and then compare the length distributions via the KS test in each of these time bins. This might work, but I don't know how well it will still capture the short time behavior of the system under study, or when I do have different starting characteristics for my time series. I would do this for all of my experimental results, and then for all of the given simulated results. Would this still be okay?
- Some sort of ARIMA model perhaps?
- Growth Curve Analysis?
- Autocorrelation of the time series, normalized to be between 0 and 1?
- Dynamic Time Warping?
Again, my overall goal is to test how well the simulated time-series reproduces what is seen experimentally. Thanks!
EDIT Here is a plot of the two different time series that I would like to compare, as an example.