3
$\begingroup$

First time poster but I've lurked here quite a bit! I need a bit of guidance with regards to what approach I should use with the below problem:

DATASET: 1 Master Log that records ~20 databases and their data transfers to one particular server (20 databases --> 1 database). Recorded daily (i.e. "Server X : 10GB, Server Y : 100KB)

DATA: Each data source has a fairly regular ingest (Server X's daily transfers won't deviate too far from 10GB, Server Y won't deviate too far from 100KB, etc). Each data source's mean daily transfer is fairly different from the other data sources (ranging from a few KB to dozens of GB depending on the server). For now, it is a univariate dataset but I have the option to add in what day-of-the-week or date if needed.

PROBLEM: I'm trying to create an anomaly detection flag script that checks to see if there was a 'break' in the transfer process. For example, if Server X normally sends 10 GB but one day it sends 200MB, that should raise a flag.

POTENTIAL SOLUTION: I'm currently testing an approach that calculates the daily mean and daily standard deviation for each Server and then using (mean +/- (2 * standard_deviation)) to check if there are outliers (as this would check to see if the daily ingest falls within 2 standard deviations thus making it a potential anomaly).

Source: https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule

COMPLICATIONS:

  1. How do I handle the anomalies that fall outside of the normal 95%? If I detect an anomaly (Server X's acceptable range is 9GB to 11GB and the daily ingest is 1 GB), do I ignore it and input the mean up to that point in the log instead? I imagine if I don't prune the data of anomalies as I go, it could warp the means and standard deviations thus making it harder to detect anomalies the more that pop up.

  2. This approach doesn't take into account that Saturdays, Sundays, and holidays have less data being transmitted compared to weekdays.

OTHER MODELS CONSIDERED: I've been reading about Time Series models, but I'm still in the research phase so I'm not extremely familiar with implementing one in a production environment. I've been brainstorming about how I could possibly use a Logistic Regression model of some sort, but I haven't been able to figure out what the features would be.

Any suggestions?

$\endgroup$
0
$\begingroup$

You might want to look at some of my posts regarding daily data ... a sore point for some but an area of excellence for others https://stats.stackexchange.com/search?q=user%3A3382++DAILY+DATA. The heart of the matter is to detect an anomaly you need a rule ... that rule is an equation not simple statistics that you have suggested.

The equation gives a prediction which generates "the probability of what was just observed" . If the probability is low an outlier is flagged. If you wish to post your data , I will introduce it to an automatic solution which you can study/evaluate for efficiency.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.