4
$\begingroup$

I am building an LSTM Autoencoder (unsupervised model) to detect anomalies in a time series dataset. The input is telemetry data from routers and I want to detect anomalies in the throughout of router. The ground truth of whether the anomaly raised by the model is actually an anomaly or not, is unavailable at the time the anomaly is raised. I would like to know what would be the correct way to measure the model performance for my use case. So far, I've come across the following metrics and techniques:

  1. Reconstruction error of autoencoder
  2. Twin-Sample Validation from this blog
  3. Cluster Analysis with two subsamples from this post

What would be the best way to measure my model's performance? Metrics and techniques other than above 3 are welcome!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

I am not sure I understand your first technique, because, AFAIK, the reconstruction error of autoencoders is what is used as the score for anomaly detection in the first place, so your first technique would mean to evaluate your anomaly detection by itself. The second and third techniques are mainly geared towards clustering, which is not always appropriate for anomaly detection, and they are also rather measuring the "variance" in the results of the techniques, not whether the results are right or wrong. You can have techniques with small or no variation in the results which are nevertheless completely wrong.

In general, you cannot prove the correctness of an unsupervised method if you don't have ground truth. You can compare it with the results of other unsupervised methods and settle for some consensus, i.e. use some kind of ensemble anomaly detection, but this doesn't really solve your problem of missing true labels.

The best would really be to somehow find some correctly labeled cases. This is mainly done by domain experts that provide positive and negative cases which your method can then be evaluated against.

In clustering, people have come up with some "internal" criteria that might evaluate the quality of a clustering, like maximizing some kind of measure for the ratio of the separation of clusters versus the cluster sizes. But those are no solid measures for the quality of the clusters and one can often find situations where those methods can be misleading.

The idea of those criteria is often that they are, themselves, clustering methods, but using them for clustering in the first place would usually scale exponentially, and thus you use the less precise, much faster methods to come up with some good candidates, and then the slow but precise methods are used to choose the winner between those candidates.

For anomaly detection, there are much fewer such internal criteria. I only know of one, the IREOS criterion, described in:

Marques, Henrique O., et al. "On the internal evaluation of unsupervised outlier detection." Proceedings of the 27th international conference on scientific and statistical database management. 2015.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks @frank. Could you please elaborate on the stuff said in the 2 paragraphs: "In clustering..." and "The idea of those criteria..."? I could not understand it fully. I found a paper that evaluated various methodologies. arxiv.org/pdf/2104.01422.pdf I haven't read it in detail but it seems they are saying that there is no metric that can help pick the right methodolgies i.e. algorithm and hyperparameters $\endgroup$
    – learnlifelong
    Commented May 6, 2022 at 1:15
  • $\begingroup$ @learnlifelong I haven't read it fully either, but it seems the point is that you have to settle for some "master" criterion and there is no way to know whether it is correct. $\endgroup$
    – frank
    Commented May 6, 2022 at 4:57
  • $\begingroup$ @learnlifelong Pertaining: "The idea of those criteria...": my point is only that many of those criteria, e.g. the Davies–Bouldin index (DBI), can be used as clustering methods themselves by simply scoring all possible partitions. Since this doesn't scale, we settle for much faster algorithms, which might be much worse than DBI. But then we use DBI at least as a tie-breaker at the end. However, while DBI seems to be quite reasonable, it might still be inappropriate. $\endgroup$
    – frank
    Commented May 6, 2022 at 5:09
  • $\begingroup$ Hi @frank, I have a question regarding model improvement. Let's say I deployed an LSTM Autoencoder (unsupervised model) to detect anomalies in a time series dataset. After sometime I created a new model, on what basis can I compare the deployed and the new model to decide whether or not to deploy the new model? $\endgroup$
    – learnlifelong
    Commented May 9, 2022 at 1:46
  • $\begingroup$ I want to add: even for clustering you should consider external evaluation metrics like rand index etc. Your clusters could otherwise be useless (some arbitrary pattern) $\endgroup$
    – Ggjj11
    Commented Nov 30 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.