First time poster but I've lurked here quite a bit! I need a bit of guidance with regards to what approach I should use with the below problem:
DATASET: 1 Master Log that records ~20 databases and their data transfers to one particular server (20 databases --> 1 database). Recorded daily (i.e. "Server X : 10GB, Server Y : 100KB)
DATA: Each data source has a fairly regular ingest (Server X's daily transfers won't deviate too far from 10GB, Server Y won't deviate too far from 100KB, etc). Each data source's mean daily transfer is fairly different from the other data sources (ranging from a few KB to dozens of GB depending on the server). For now, it is a univariate dataset but I have the option to add in what day-of-the-week or date if needed.
PROBLEM: I'm trying to create an anomaly detection flag script that checks to see if there was a 'break' in the transfer process. For example, if Server X normally sends 10 GB but one day it sends 200MB, that should raise a flag.
POTENTIAL SOLUTION: I'm currently testing an approach that calculates the daily mean and daily standard deviation for each Server and then using (mean +/- (2 * standard_deviation)) to check if there are outliers (as this would check to see if the daily ingest falls within 2 standard deviations thus making it a potential anomaly).
How do I handle the anomalies that fall outside of the normal 95%? If I detect an anomaly (Server X's acceptable range is 9GB to 11GB and the daily ingest is 1 GB), do I ignore it and input the mean up to that point in the log instead? I imagine if I don't prune the data of anomalies as I go, it could warp the means and standard deviations thus making it harder to detect anomalies the more that pop up.
This approach doesn't take into account that Saturdays, Sundays, and holidays have less data being transmitted compared to weekdays.
OTHER MODELS CONSIDERED: I've been reading about Time Series models, but I'm still in the research phase so I'm not extremely familiar with implementing one in a production environment. I've been brainstorming about how I could possibly use a Logistic Regression model of some sort, but I haven't been able to figure out what the features would be.