Please forgive my naive questions.
I have data collected on different production servers about the performance of our system.
Basically, approximately once a minute a numerical value is recorded for each of a set of interesting aspects of our system. My task is raising some kind of alarm when one of this value is unusual.
My approach so far has been something like "I have to find outliers". All of my data show some seasonality, depending on the time of day, the day of week and the month of the year. Some of the data are counters, some others are proportions.
My first attempt has been to STL function in R for one of this counters for a week but STL returns something along "your data aren't periodical or there aren't enough periods".
More naively, I took the average and stdev of the data from all the time ranges around the same time of day, for the same weekday and month (example: if now it's 5:30 of a November Friday, I consider all the data between of all the November Fridays in the past, betweem 5:00 and 6:00 ).
After this, I consider unusual all the data that have a distance from the mean more than 2sigma ( I raise a warning if the zScore is between 1 and 2 and an alarm if greater than 2).
This somehow works but of course it is very very inaccurate as none of my data seems to be normally distributed.
The second step was to find one set of data that seemed bell-shaped and trying to see if it really was. I tried to fit it with several distribution (normal, logNormal, Poisson, logistic, gamma, Weibull) using fitdist. Then I compared all these by superimposing these distribution on several graphs (CDFs, Q-Q, P-P and theoretical densities over the Histogram) and it seems "logistic" is the distribution that best approximate this set of data.
Now, I would like to find the range outside of which my data have to be considered as unusual as they would be with 1.96 sigma if the distribution were normal (2.5% on each side). I guess there is a R function that gives me that but my first question is:
1) is it correct to exlude data in this way? That is: whatever distribution I am able to find for my data, to exlude the outliers do I always have to use a (CDF?) function specific for that distribution that tells me where the 2.5% left range ends and where the 97.5% right range begins?
Since I have to automate this job, I can't always look at my hundreds of possible data and guess their distribution by looking at 4 graphs for each. I need something automated. Therefore, in my case, I ran a gofstat for all my distribution and it does seems that the lower values for the 3 statistics presented is always the one for the logistic.
2) In order to make it automatic, can I always assume that the best distribution, among those that I know, is the one that gets lower values in these tests? Should I just authomatically try to fit all possible distributions until a best match is found? This seems too random to me.
3) Please explain what would be wrong in doing this: - I divide the range of the data in small ranges. The number of ranges should be such that at least some of the ranges receives at least a certain percentage of data. - I save what percentage of data that falls in each of the small ranges. - If a new data point falls in a range with a low percentage, I raise an alarm. In this way I can manage whatever distribution there may be.
Among the things that I read is "http://blog.cloudera.com/blog/2015/12/common-probability-distributions-the-data-scientists-crib-sheet/"
I read the "Similar Questions" on the right but none is addressing the fact that I am more interested in the possible automatization of the process. I don't need to fit one single set of data with the absolutely best distribution ever: what I really need is , for many many different set of data, to be able to autmatically (and quickly) label some points as potentially alarming.
I am working with RStudio and, as it is probably appearent, I am quite new to all this. I will not easily understand extremely technical answers (but I'll do my best, in case)