Daily Data Transfer Logs - Anomaly Detection

Question

First time poster but I've lurked here quite a bit! I need a bit of guidance with regards to what approach I should use with the below problem:

DATASET: 1 Master Log that records ~20 databases and their data transfers to one particular server (20 databases --> 1 database). Recorded daily (i.e. "Server X : 10GB, Server Y : 100KB)

DATA: Each data source has a fairly regular ingest (Server X's daily transfers won't deviate too far from 10GB, Server Y won't deviate too far from 100KB, etc). Each data source's mean daily transfer is fairly different from the other data sources (ranging from a few KB to dozens of GB depending on the server). For now, it is a univariate dataset but I have the option to add in what day-of-the-week or date if needed.

PROBLEM: I'm trying to create an anomaly detection flag script that checks to see if there was a 'break' in the transfer process. For example, if Server X normally sends 10 GB but one day it sends 200MB, that should raise a flag.

POTENTIAL SOLUTION: I'm currently testing an approach that calculates the daily mean and daily standard deviation for each Server and then using (mean +/- (2 * standard_deviation)) to check if there are outliers (as this would check to see if the daily ingest falls within 2 standard deviations thus making it a potential anomaly).

Source: https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule

COMPLICATIONS:

1. How do I handle the anomalies that fall outside of the normal 95%? If I detect an anomaly (Server X's acceptable range is 9GB to 11GB and the daily ingest is 1 GB), do I ignore it and input the mean up to that point in the log instead? I imagine if I don't prune the data of anomalies as I go, it could warp the means and standard deviations thus making it harder to detect anomalies the more that pop up.

2. This approach doesn't take into account that Saturdays, Sundays, and holidays have less data being transmitted compared to weekdays.

OTHER MODELS CONSIDERED: I've been reading about Time Series models, but I'm still in the research phase so I'm not extremely familiar with implementing one in a production environment. I've been brainstorming about how I could possibly use a Logistic Regression model of some sort, but I haven't been able to figure out what the features would be.

Any suggestions?

Answer 1

You might want to look at some of my posts regarding daily data ... a sore point for some but an area of excellence for others https://stats.stackexchange.com/search?q=user%3A3382++DAILY+DATA. The heart of the matter is to detect an anomaly you need a rule ... that rule is an equation not simple statistics that you have suggested.

The equation gives a prediction which generates "the probability of what was just observed" . If the probability is low an outlier is flagged. If you wish to post your data , I will introduce it to an automatic solution which you can study/evaluate for efficiency.

Answer 2

What you are proposing to create are called statistical control or Shewhart charts. There are many references, I would recommend Chapter 14 of Box's book "Statistics for Experimenters".

In response to your questions:

Regarding 1., if you can't pin down a reason why the 1GB transfer is abnormal, then there is no reason to exclude it, since it comes up as part of the "normal operation" of the process. This implies that the mean and sd you will use should take that value into account, otherwise you will underestimate the variability. Of course, you may want your chart to forget about values very far in the past, and therefore take only the mean and sd based on the last $N$ values, for some well-chosen $N$. If, on the other hand, you can flag the value of 1GB as erroneous, then you need not consider it in the computation of your mean and sd.

Regarding 2., a simple adjustment would be to compare each measurement with the mean of the corresponding category (weekdays vs not). You could compute two means and two sds, one for each category, and only compare like with like. A better model would use a time series model (these are briefly covered in the Box book).