1
$\begingroup$

Let me first explain what I'm trying to do:

An autoencoder is first trained on regular data (no anomalies), then the decoder is discarded. Outputs from the encoder using overall data (data with and without anomalies) is clustered (using HDBSCAN) and the clusters are manually examined. The hypothesis is that anomalies will not become a part of the core clusters or become outliers. There's a few papers citing good results from this (just search "autoencoder anomaly detection clustering").

However, my autoencoder yields very low accuracy (<20%) and high loss. This concerns me but at the same time I question whether if this actually matters since I only care about the latent space produced by the encoder. Does anyone have inputs on this?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ When you say "accuracy" what are you measuring, specifically? Usually "accuracy" in a machine learning setting means the proportion of correct predictions, but I don't understand how you would apply that in this setting. As an aside, isolation forest might be a better choice (either applied to the encoded data or to the raw data) because it is explicitly an "outlier" detector. $\endgroup$
    – Sycorax ♦
    Mar 30, 2022 at 19:28
  • $\begingroup$ accuracy of the autoencoder only. Not including the clustering. $\endgroup$
    – bli00
    Mar 30, 2022 at 19:40
  • $\begingroup$ So reconstruction accuracy $\endgroup$
    – bli00
    Mar 30, 2022 at 19:41

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.