Model performance when ground truth is not available

Question

I am building an LSTM Autoencoder (unsupervised model) to detect anomalies in a time series dataset. The input is telemetry data from routers and I want to detect anomalies in the throughout of router. The ground truth of whether the anomaly raised by the model is actually an anomaly or not, is unavailable at the time the anomaly is raised. I would like to know what would be the correct way to measure the model performance for my use case. So far, I've come across the following metrics and techniques:

1. Reconstruction error of autoencoder
2. Twin-Sample Validation from this blog
3. Cluster Analysis with two subsamples from this post

What would be the best way to measure my model's performance? Metrics and techniques other than above 3 are welcome!

Answer 1

I am not sure I understand your first technique, because, AFAIK, the reconstruction error of autoencoders is what is used as the score for anomaly detection in the first place, so your first technique would mean to evaluate your anomaly detection by itself. The second and third techniques are mainly geared towards clustering, which is not always appropriate for anomaly detection, and they are also rather measuring the "variance" in the results of the techniques, not whether the results are right or wrong. You can have techniques with small or no variation in the results which are nevertheless completely wrong.

In general, you cannot prove the correctness of an unsupervised method if you don't have ground truth. You can compare it with the results of other unsupervised methods and settle for some consensus, i.e. use some kind of ensemble anomaly detection, but this doesn't really solve your problem of missing true labels.

The best would really be to somehow find some correctly labeled cases. This is mainly done by domain experts that provide positive and negative cases which your method can then be evaluated against.

In clustering, people have come up with some "internal" criteria that might evaluate the quality of a clustering, like maximizing some kind of measure for the ratio of the separation of clusters versus the cluster sizes. But those are no solid measures for the quality of the clusters and one can often find situations where those methods can be misleading.

The idea of those criteria is often that they are, themselves, clustering methods, but using them for clustering in the first place would usually scale exponentially, and thus you use the less precise, much faster methods to come up with some good candidates, and then the slow but precise methods are used to choose the winner between those candidates.

For anomaly detection, there are much fewer such internal criteria. I only know of one, the IREOS criterion, described in:

Marques, Henrique O., et al. "On the internal evaluation of unsupervised outlier detection." Proceedings of the 27th international conference on scientific and statistical database management. 2015.