What sort of analysis method is most appropriate for computer monitoring data that includes samples at a given time?

Question

I'm interested in what sort of modelling is best suited for data that includes samples collected at different times.

My use case is to do with computer monitoring, so for the initial case assume that the data has 2 parameters: CPU load and the time of the sample. In the future I want to scale this to look at many features.

The CPU parameter is expected to vary across a given day (and perhaps a week): for example, users will come in to work and log into their computers in the morning and perform some complicated queries first thing, which will result in a spike of CPU that will last a while. Thereafter, CPU might plateau or decay a bit and at some point, people will go home and CPU will decrease.

I am interested in understanding both absolute anomalies (e.g. CPU > 90% irrespective of time, meaning perhaps you need a new machine, as well as time-based anomalies, e.g. 9am on Monday it's normal for CPU load to be high, but not 5pm on a Sunday and thus there's something odd that's going on).

I am interested in the possibilities that a Bayesian approach might offer, as I have no fixed opinions on what the data will look like and what "normality" will look like or what probabilities are associated with seeing a particular event. I'd like to have my model give me a probability that expresses the likelihood of seeing a particular event and to update the model over time to reflect the new posterior based on prior events.

Are there particular approaches that I should look at or ideas as to how to best solve this? Any particular papers that I could go to read? My maths is pretty reasonable but I'm not a statistician, I'm a software engineer, so I'd prefer to have resources that are not impenetrable to those without a PhD in maths please.

Answer 1

Without a clear idea of what "normal" data looks like, there is no way to figure out "anomalies", since these are by definition events that deviate from some kind of norm. So the first step is to re-evaluate what you mean by "normal" and "anomaly". From reading your post, I have the feeling you actually have some points in there in that direction, but those are not clear enough yet.

Now to define anomalies, you don't necessary have to manually set the criterion. You can always set a criterion based on previous data. For example you could define an anomaly as "CPU-load is much higher than the average CPU-load on previous comparable days. Still that definition includes, that you perceive the historic data as "normal" and define anything that deviates from that as "anomaly". There are many more approaches to those definitions, you could take and those then are crucial in defining your model.

If you have a definition based on the history, the simplest way for anomaly detection can be based on regression analysis. The statistical interpretation of regression analysis is that you are looking for parameters beta for which $y = \beta_0 + \sum \beta_i*x_i + \epsilon$ where $\epsilon$ is an error term following some kind of distribution that should be minimized. Then you can both estimate the parameter $\beta_i$ as well as the parameters of the error term. That can easily be done by optimizing with log-likelihood as the criterion (there are good implementations out there in almost any language, that will also give you the distribution parameters). Bayesian estimation also works here, but is not really needed if you only need the best estimates and also usually takes more computing time.

No once you have the model including distribution parameters, you can easily plug in new parameters that you gathered while your system is running. If you then compare the prediction of the model to the actual value, you get an error for that time. Since you got the distribution of that error, you can use that to get the probability of such an error and this will tell you, if that time can be considered an anomaly.

Ok, I guess we'll need an example to be more clear. Let us take the definition: "A CPU-Load is unusually high, if it deviates strongly from the historic load at a the same hour of the previous days". Then you could define your $x_i$ with $i$ in 1 to 24 as 1 if it currently is that time and 0 otherwise (dummy coding). You fit your model with the $\epsilon$ distributed normally with mean zero and unknown variance $\sigma^2$ and you may get back $\beta_0=0.1$, $\beta_15=0.5$ and $\sigma^2=0.01$. If you then sample an average load of 0.7 at the next day at 15:00, you would get $y_{pred}=0.1+0.5=0.6$ and therefor $\epsilon=0.6-0.1=-0.1$. If you plug that into your normal you get $N(\epsilon<-0.1|\mu=0,\sigma^2=0.01)=0.16$, so a 16% chance of this just being a normal random fluctuation. Usually a cutoff of 5% or 1% chance is taken to determine if one should consider something unusual. So in this case, the deviation is not unusual.

Now the model in the example is very simplistic and probably not up to the task. You may find it works better, if you assume some other kind of error distribution, define other ways to encode your reference group etc. All this goes in the direction in determining what you define as "normal", a step that just cannot be avoided.