# Comparing anomaly detection algorithms

My question is how can different anomaly detection algorithms be compared for my specific dataset. Essentially, I have multivariate timeseries (physical quantities such as temperature, pressure) and a label in one of the time stamps saying that my physical system reached some critical state (e.g. had to shutdown, failed, etc). So the issue is that I don't really have a nice "labels", just an indication of a failure at some point.

The task I am working on is to "come up" with a measure of how good my anomaly detection algorithms are, given this not so nice dataset I have.

So my idea is: there are two essential factors - detecting anomalies and not raising false alarms.

So make a standardized dataset for evaluation purposes containing both abnormal and normal functioning of my system.

The cost function, which I want to minimize, could be given by: $Cost = w_1*\sum_i^{sample containing failure} 1[i^{th} \mbox{ failure not detected}] +w_2 \sum_i^{normal samples} 1[i^{th} \mbox{ false positives}]$.

For simplicity I just define the latter as a simple indicator function and the first as:

$$1[\mbox{event not detected}] = \left\{\def\arraystretch{1.2}% \begin{array}{@{}c@{\quad}l@{}} 0 & \text{if anomaly raised within n hours before the time labelled as shutdown}\\ 1 & \text{otherwise}\\ \end{array}\right.$$

The reason there’s a n hour window before the event is because there's the assumption we could already see some anomaly before the shutdown and why false positives shouldn’t be considered on the time series containing events is due to the fact that there’s no guarantee that the time prior to n h before the labelled timestamp event doesn’t contain anomalies, thus, the algorithm could be correct in flagging abnormal behavior.

My question is does this make sense? Do you think this would give a good impression of the quality of my algorithms? Otherwise, how would you go about comparing algorithms, given that the data is not clearly labelled? I considered using some standardized time series datasets from http://www.cs.ucr.edu/~eamonn/time_series_data/ but there's interest in seeing whether it actually works in this messy dataset.

Further info: the timeseries are not stationary (but can be broken into sub stationary time series). The anomaly detection methods I'm comparing are One class svm, AAKR and some sort of moment shift tracker.

Thanks a lot for your help, any feedback, resources or counter examples to why this is a bad way to compare algorithms is super appreciated.