I have several (~10) models which all predict time series results for a specific problem (hourly heat loss from a building over a year). I do not have any measured data or truth standard results to compare the absolute accuracy of each model's prediction. I am looking for a way to quantitatively compare each time series to the others to give some indication of agreement among the models, or to identify outlying results.

Do such metrics exist? If so, what metrics would be useful for me?

Here is a quick sketch of what the results might look like (each color represents the results from a different model):

enter image description here

  • $\begingroup$ Looks like they differ in variability and mean. Why now fit a quadratic curve to each to represent the mean and then look at the residuals to examine variability? Of course, if you know what the mean is for each time period, then you should use that curve. $\endgroup$ – user31668 Sep 15 '14 at 16:04

What you are doing is (part of) time series clustering. (Since you don't have the actuals, you are doing a kind of unsupervised learning.) As in clustering other things, this is a two-step approach:

  1. Define a distance or metric on the objects you are looking at (this you explicitly mention)
  2. Cluster the objects based on distances (this you do not mention explicitly - but it is kind-of-sort-of implied in your question about "identifying outlying results")

One advantage is that we can now divide our problem into two logically separate pieces: finding a distance between time series, and clustering using this distance.

As to distances or dissimilarities between time series, I have had good results with potentially renormalizing each series so that it sums to 1 (i.e., turns into a probability density) over time and then calculating Hellinger distances. Given that you presumably do care about the y axis, you probably don't want to rescale in your application.

As to clustering, I'd suggest you look into DBSCAN. It doesn't require you to prespecify the number of clusters as the better known k-means algorithm (of course, you still have one or two tuning parameters you'll have to think about), and more importantly, it can identify outliers, in contrast to k-means, which will assign every series to a cluster. Indeed, the "N" in DBSCAN stands for "noise", which is DBSCAN's name for outliers.

That said, there is quite a literature on time series clustering, so you may want to read up a bit on all this. Here are a couple of examples using R. Here is a survey.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.