7
$\begingroup$

Let's assume an analytical model predicts an epidemic trend over time, i.e. number of infections over time. We also have a computer simulation results over time to verify the performance of the model. The goal is to prove the simulation results and predicted values of the analytical model (which are both a time series) are statistically close or similar. By similarity I mean the model predicts the values close to what simulation is providing.

Background: Researching around this topic, I came across the following posts:

  1. https://stackoverflow.com/questions/13835924/similarity-of-trends-in-time-series-analysis

  2. How to statistically compare two time series?

Both discussions suggest three approaches, where I am interested in two of them basically:

(1). Use of ARIMA; (2). Use of Granger test

For the first suggested solution, this is what has been written there in regards to ARIMA, in (1):

Run ARIMA on both data sets. (The basic idea here is to see if the same set of parameters (which make up the ARIMA model) can describe both your temp time series. If you run auto.arima() in forecast (R), then it will select the parameters p,d,q for your data, a great convenience.

I ran auto.arima on the simulation values and then ran forecast, here are the results:

ARIMA(2,0,0) with zero mean     

Coefficients:
         ar1      ar2
      1.4848  -0.5619
s.e.  0.1876   0.1873

sigma^2 estimated as 121434:  log likelihood=-110.64
AIC=227.27   AICc=229.46   BIC=229.4

I ran auto.arima on predicted model values and then forecast. This is the result of the predicted model:

ARIMA(2,0,0) with non-zero mean 

Coefficients:
         ar1      ar2  intercept
      1.5170  -0.7996  1478.8843
s.e.  0.1329   0.1412   290.4144

sigma^2 estimated as 85627:  log likelihood=-108.11
AIC=224.21   AICc=228.21   BIC=227.05

Question 1 What are the values that need to be compared to prove that the two series are similar especially the trend over time?

Regarding the second suggested option, I have read about it and found that Granger test is usually used to see if the values of series A at time t can predict the values of Series B at time t+1.

Question 2 Basically, in my case I want to compare the values of time series A and B at the same time, how this one is relevant to my case then?

Question 3 Is there any available method can be used to prove that the trend of two time-series over time is similar?

FYI. I saw another method which is using Pearson Correlation Coefficient and I could follow the reasoning there. Moreover, verifying analytical models with simulations has been widely used in the literature. see:

  1. Spatial-Temporal Modeling of Malware Propagation in Networks Modeling
  2. Modeling and Simulation Study of the Propagation and Defense of Internet Email Worm
$\endgroup$
  • $\begingroup$ This question has quite a lot of hypotheticals. Could you re-phrase it to what you actually are trying to do? A lot depends on what you mean by "similar". How was the simulation generated, if not already by a model? In which case why can't you analytically compare the two models? If they are differ, how do you know that says anything about the validity of the first model, rather than of the simulation? $\endgroup$ – Peter Ellis Sep 13 '15 at 2:11
  • $\begingroup$ Thanks Peter for your comment. The final goal is to say the model predicts the values close to what simulation is giving. That is what I meant by similarity. I have also edited the question to ensure other readers will not get confused. I did not quite understand what do you mean by "why can't you analytically compare the two models?", what are the methods to do so? and regarding your last comment, to the best of my knowledge it is common to verify analytical models with simulations. $\endgroup$ – Moe Sep 13 '15 at 2:50
  • $\begingroup$ What I'm puzzling about is how you simulated the results; presumably with a second model (I don't know how you could simulate them without a model). Unless you mean a physical simulation? I'd been presuming you were generating the simulation with a computer program ie a model. In which case who is to say it is better than the first model? $\endgroup$ – Peter Ellis Sep 13 '15 at 2:54
  • $\begingroup$ Verifying models using simulations are common in computer malware that can be classified as epidemiological infections. For instance, see the following references: Spatial-Temporal Modeling of Malware Propagation in Networks Modeling and Simulation Study of the Propagation and Defense of Internet Email Worm and Assuming this is a correct way of verifying models, the questions is how to prove two time series are statistically close to each oter? $\endgroup$ – Moe Sep 13 '15 at 13:56
  • $\begingroup$ Judging from the ARIMA results, the simulation values are for a stationary time series that fluctuates in the interval [-10, 10] (approximately). This doesn't look like a number of infections; or even a number of new infections. Have you transformed it, or taken differences twice, or something, before fitting auto.arima? $\endgroup$ – Peter Ellis Sep 13 '15 at 21:42
8
$\begingroup$

Here's what I understand the situation is. You have one model, which you call your simulation, that you are confident generates a set of data that accurately represents what will actually happen in the epidemic. For some reason (presumably because it's expensive or slow to build and run, or there's theoretical interest in a simple equation that generates similar results to the complex model), you have an alternative model (the one you call a model) which can also generate a set of data, and you want to check if the version generated by this model is close to the version generated by the known-good model.

I'm also presuming that each time either of the models generates data, it generates a similar and pretty regular trend to other times. Otherwise (for example, if there's a random "take off" moment where the series suddenly breaks) there's another big complication.

First, the method of comparing parameters from an auto-fit ARIMA is a bad one (I'm guessing the reason that answer you linked to survived is that it is on Stack Overflow rather than Cross-Validated, where the statistical problems would have been picked up probably). The reason is that the same time series can get good fits with quite different combinations of auto-regressive and moving average values. There's no obvious way to look at the "similarity" of two different ARIMAs - ones that look very different may in fact be similar. As @IrishStat says in his answer to the second question you linked to, you could construct an F-test of a common set of parameters for both models, but that would require something quite a bit more complex than auto.arima(). And even then you might find that they don't have common parameters, but deliver similar predictions of the trend which is what you are actually interested in, rather than the details of the ARMA process that is generating some of the random noise around the trend.

So what would I recommend instead? It sounds like you aren't worried about the small fluctuations but only the overall trend. I would compare a smoothed version of the trend of each dataset, and start by making a visual comparison. In the case you've got, this shows that they're definitely not the same time series; one of them hovers around 1478, the other around zero, and that's good enough for me. But if there were some ambiguity, I would probably sum the squares or absolute values of the difference between the two smoothed series and determine if that was close enough, for some arbitrarily chosen meaning of "close enough" which in the end will have to depend on your domain, and the costs of being wrong. Definitely I'd start with the graphic.

If you want a more objective benchmark, I would try running both simulations multiple times and seeing how much difference (sum of squares or absolute differences) there is between different instances of the same simulation, and comparing that to the inter-simulation differences. If they're the same, that shows that you can't tell which model produced the simulation. If they're different, you still have to make a judgement call about how different is too much, but you'll have some numbers to help you.

enter image description here

While fitting ARIMA models is a bad idea for identifying similarity in trends, it's a good way to let me generate some data, so pasted below is how I did that. I'm guessing something's wrong with the data - maybe you fit the ARIMA model to a transformed or differenced version of the data, in which case you might want to go the next step of quantifying the difference between the two trends.

library(forecast)
library(ggplot2)
library(tidyr)
library(dplyr)

# generate some data
good_model <- arima.sim(model = list(ar = c(1.4848, -0.5619)), n = 1000)
test_model <- arima.sim(model = list(ar = c(1.5170, -0.7996)), n = 1000) + 1478


combined <- data.frame(good = good_model, test = test_model, time = 1:1000)  %>%
 gather(variable, value, -time) %>%
 mutate(value = as.numeric(value))

 ggplot(combined, aes(x = time, colour = variable, y = value)) +
    geom_line(alpha = 0.5) +
    geom_smooth(se = FALSE, size = 2) +
    theme_minimal()

Edit

I blogged about this at http://ellisp.github.io/blog/2015/09/20/timeseries-differences , basically just exploring how you might use simulation brute force to determine if two models are similar. However, I reach the conclusion that you still need a (probably) subjective decision on a cost function - obviously you're two methods will be different, but how different are you prepared to put up with?

$\endgroup$
  • $\begingroup$ About comparing ARMA models: Even when we cannot compare coefficients, we could use the impulse response functions, spectral densities or autocorrelation functions as a measure for the closeness of two processes. This would also work if we only compare processes and don't have the same events or predictions in both cases. $\endgroup$ – Josef Sep 14 '15 at 19:42
  • $\begingroup$ Thanks, it might help, but even then you can have identical acfs and two very different processes eg see ellisp.github.io/blog/2015/09/19/timeseries-same-acf . I'm pretty sure the answer requires multiple runs of each data generating black box and create a similarity statistic of some sort. $\endgroup$ – Peter Ellis Sep 18 '15 at 23:43
  • $\begingroup$ Interesting post and I don't disagree. A few points 1) do we want to compare processes or realizations: If we can use the same shocks to the system (innovations), then we can compare the forecasts directly and this might be the main answer to the original question. If we don't have comparable realization, then we can only compare processes given some assumption on the innovation process. I was thinking of the covariance stationary linear model case (pure ARMA) where the impulse response function (moving average representation) summarizes the forecasts and impact of a shock. $\endgroup$ – Josef Sep 19 '15 at 4:13
  • $\begingroup$ 2) an ARMA process only defines the response of a system to shock or innovations. This does not include the variance of the innovation process itself. Similar autocorrelation is not sufficient to describe all features of the data. As a correlation it is based on standardized data, we still need means and variances. Even autocovariance functions wouldn't have any information about the mean. However, the impulse response function would summarize the system response after taking out differencing, drift, means and deterministic or exogenous effects. $\endgroup$ – Josef Sep 19 '15 at 4:28
  • $\begingroup$ 3) (hedging my bets) If we leave the linear ARMA framework and also consider Markov switching, regime switching or other nonlinear models, then comparing the implication of models is more difficult. I don't know enough about those cases to say much. I'm thinking of those because it is not obvious to me that a two regime model, with one state being without more than a few outlier infection cases and a second state with being in an epidemic is not the better reference model than a (linear) ARMA process. $\endgroup$ – Josef Sep 19 '15 at 4:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.