I am attempting to find abnormal usage patterns in a billing system.
Usage looks like this:
ID Datetime Quantity A 2016-01-01 10 A 2016-01-01 50 A 2016-01-02 20 A 2016-01-02 40
Usage has some pretty standard characteristics. It is high during the weekdays, and falls during the weekends. On holidays it falls. Fridays are a tad lower than Thursdays. Therefor, I cannot just calculate the average and standard deviation without slicing up the timestamp to account for these changes.
I have split the timestamp into the following metrics:
- Day of Month
- Day of Week
- Is Holiday
id month day_of_month week day_of_week is_holiday quantity A Jan 1 1 Monday false 10 A Jan 1 1 Monday True 50
Alternatively, I could simply use two metrics:
I've then computed the average and standard deviation across all these metrics for each ID since the beginning of time.
So let's take the following scenario: Today is a Monday, in the 7th week of the year, in the 2nd month of the year, on the 4th day of the month, and it is not a holiday. I've already computed the following for these metrics:
Metric Avg Stddev Monday 2500 800 Not Holiday 2000 1000 7th week 2200 600 2nd month 2400 300 4th day of month 1900 800
Is there any way to take these numbers and compute a "target average" of 2245.5 and "target stddev" of 758.2, to which I can compare against today's usage?
Otherwise, I could use the following rules, for example:
- If the usage falls outside a stddev for just one metric, flag it as abnormal
- If the usage falls outside of a stddev for ALL metrics, flag it as abnormal
Is there a standard way of detecting anomalies given all the ways I can slice a timestamp?