How to identify outliers in server uptime performance data?

Question

I have a python script that creates a list of lists of server uptime and performance data, where each sub-list (or 'row') contains a particular cluster's stats. For example, nicely formatted it looks something like this:

-------  -------------  ------------  ----------  -------------------
Cluster  %Availability  Requests/Sec  Errors/Sec  %Memory_Utilization
-------  -------------  ------------  ----------  -------------------
ams-a    98.099          1012         678          91
bos-a    98.099          1111         12           91
bos-b    55.123          1513         576          22
lax-a    99.110          988          10           89
pdx-a    98.123          1121         11           90
ord-b    75.005          1301         123          100
sjc-a    99.020          1000         10           88
...(so on)...

So in list form, it might look like:

[[ams-a,98.099,1012,678,91],[bos-a,98.099,1111,12,91],...]

My question:

• What's the best way to determine the outliers in each column? Or are outliers not necessarily the best way to attack the problem of finding 'badness'?

In the data above, I'd definitely want to know about bos-b and ord-b, as well as ams-a since it's error rate is so high, but the others can be discarded. Depending on the column, since higher is not necessarily worse, nor is lower, I'm trying to figure out the most efficient way to do this. Seems like numpy gets mentioned a lot for this sort of stuff, but not sure where to even start with it (sadly, I'm more sysadmin than statistician...). When I asked over at Stack Overflow, someone mentioned using numpy's scoreatpercentile function and throw out anything over 99th percentile - does that seem like a good idea?

(Cross-posted from stackoverflow, here)

Answer 1

Based on the way you phrase the question

are outliers not necessarily the best way to attack the problem of finding 'badness'?

It is not clear that you are looking for outliers. For example, it seems that you are interested in machines performing above/below some threshold.

As an example, if all of your servers were at 98 $\pm$ 0.1 % availability, a server at 100% availability would be an outlier, as would a server at 97.6% availability. But these may be within your desired limits.

On the other hand, there may be good reasons apriori to want to be notified of any server at less than 95% availability, whether or not there are one or many servers below this threshold.

For this reason, a search for outliers may not provide the information that you are interested in. The thresholds could be determined statistically based on historical data, e.g. by modeling error rate as poisson or percent availability as beta variables. In an applied setting, these thresholds could probably be determined based on performance requirements.

Answer 2

A simple way to find anomalous servers would be to assume they are identically distributed, estimate the population parameters, and sort them according to their likelihoods, ascending. Column likelihoods would be combined either with their product or their minimum (or some other T-norm). This works pretty well as long as outliers are rare. For outlier detection itself, stable population parameters are usually estimated iteratively by dropping any discovered outliers, but that's not vital as long as you're manually inspecting the list and thereby avoiding thresholding.

For the likelihoods, you might try Beta for the proportions and Poisson for the rates.

As pointed out by David, outlier detection is not quite the same as reliability analysis, which would flag all servers that exceed some threshold. Furthermore, some people would approach the problem trough loss functions - defining the pain you feel when some server is at 50% availability or 500 error rate, and then rank them according to that pain.

Answer 3

Identifying a given data point as an outlier implies that there is some data generating process or model from which the data are expected to come from. It sounds like you are not sure what those models are for the given metrics and clusters you are concerned about. So, here is what I would consider exploring: statistical process control charts.

The idea here would be to collect the

• %Availability
• Requests/Sec
• Errors/Sec
• %Memory_Utilization

metrics for each of your clusters. For each metric, create a subset of the data that only includes values that are "reasonable" or in control. Build the charts for each metric based on this in-control data. Then you can start feeding live data to your charting code and visually assess if the metrics are in control or not.

Of course, visually doing this for multiple metrics across many clusters may not be feasible, but this could be a good way to start to learn about the dynamics you are faced with. You might then create a notification service for clusters with metrics that go out of control. Along these lines, I have played with using neural networks to automatically classify control chart patterns as being OK vs some specific flavor of out-of-control (e.g. %availability trending down or cyclic behavior in errors/sec). Doing this gives you the advantages of statistical process control charts (long used in manufacturing settings) but eases the burden of having to spend lots of time actually looking at charts, since you can train a neural network to classify patterns based upon your expert interpretation.

As for code, there is the spc package on pypi but I do not have any experience using this. My toy example of using neural networks (naive Bayes too) can be found here.