1
$\begingroup$

I am attempting to find abnormal usage patterns in a billing system.

Usage looks like this:

ID    Datetime     Quantity
A     2016-01-01   10
A     2016-01-01   50
A     2016-01-02   20
A     2016-01-02   40

Usage has some pretty standard characteristics. It is high during the weekdays, and falls during the weekends. On holidays it falls. Fridays are a tad lower than Thursdays. Therefor, I cannot just calculate the average and standard deviation without slicing up the timestamp to account for these changes.

I have split the timestamp into the following metrics:

  • Month
  • Day of Month
  • Day of Week
  • Week
  • Is Holiday

E.g.,

id  month  day_of_month week day_of_week is_holiday  quantity
A   Jan    1            1    Monday      false       10
A   Jan    1            1    Monday      True        50

Alternatively, I could simply use two metrics: is_weekend and is_holiday.

I've then computed the average and standard deviation across all these metrics for each ID since the beginning of time.

So let's take the following scenario: Today is a Monday, in the 7th week of the year, in the 2nd month of the year, on the 4th day of the month, and it is not a holiday. I've already computed the following for these metrics:

Metric           Avg   Stddev
Monday           2500  800
Not Holiday      2000  1000
7th week         2200  600
2nd month        2400  300
4th day of month 1900  800

Is there any way to take these numbers and compute a "target average" of 2245.5 and "target stddev" of 758.2, to which I can compare against today's usage?

Otherwise, I could use the following rules, for example:

  • If the usage falls outside a stddev for just one metric, flag it as abnormal
  • If the usage falls outside of a stddev for ALL metrics, flag it as abnormal

Is there a standard way of detecting anomalies given all the ways I can slice a timestamp?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.