Skip to main content
Source Link

Model performance when ground truth is not available

I am building an LSTM Autoencoder (unsupervised model) to detect anomalies in a time series dataset. The input is telemetry data from routers and I want to detect anomalies in the throughout of router. The ground truth of whether the anomaly raised by the model is actually an anomaly or not, is unavailable at the time the anomaly is raised. I would like to know what would be the correct way to measure the model performance for my use case. So far, I've come across the following metrics and techniques:

  1. Reconstruction error of autoencoder
  2. Twin-Sample Validation from this blog
  3. Cluster Analysis with two subsamples from this post

What would be the best way to measure my model's performance? Metrics and techniques other than above 3 are welcome!