I have a multitude of daily time series representing the volume of a certain product arriving per day at a station. There are as many time series as their are stations, and they each look like the following:

date             volume
2020-01-01       18000        # truck arrival on this day
2020-01-02       0            # no truck
2020-01-03       0            # no truck
2020-01-04       12000        # truck arrival
2020-01-05       0            # no truck
2020-01-06       0            # etc.
2020-01-07       0
2020-01-06       21000
2020-01-07       11000
...              ...

In reality, each time series spans multiple years.

What I'm trying to do is to build a system to give a score of "unusualness" for each time series, on each day, based on the arrivals observed during the previous days/weeks, such as if the past weeks have seen much more (or much less) cumulative arrivals than past rolling year, give it a high score.

I've tried many different ways, mostly based on rolling window aggregations, that would be quite long to explain here, but one consistent issue is the following:

  • depending on the parameters I set:
    • either the time-series with a small overall volume tend to be scored high, because the relative variations matter more
    • or the time-series with a big overall volume tend to be score high, because the absolute variations are big
    • but I'm struggling to find a way to marry the two and give an adequate, balanced weighing between "overall-big" and "overall-small" time-series

Also, the methods I've tried all seem a bit "hacky" and akin to throwing duct-tape at the problem. I'd be ideally looking for something cleaner than playing around with nested rolling aggregations and hoping for the best.

Below is a visual example of the kind of result I'm after:

enter image description here

Without necessarily going into details (I'm obviously not asking for a full solution), what would be some valid, state-of-the-art approaches to deal with that kind of problems?

  • 1
    $\begingroup$ I found this thesis (from 2016) to give a good coverage of different approaches, including LSTMs: mars.gmu.edu/handle/1920/10250 $\endgroup$ Commented Jul 9, 2020 at 18:40


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.