Autoencoder reconstruction error threshold

Question

I have a set of signals on which I have to implement an anomaly detection algorithm. The data is split among a reference period (i.e. last 3 months) and a test period (i.e. last week). I've already built an autoencoder model which is trained (and validated) with the data on the reference period. Then the test period is scored.

I would like to know how to make an "indicator" (single number) of how the two distribution look alike, or how "anomalous" is the second distribution compared to the first one.

I have calculated the reconstruction error in the training set and in the test set for each point (calculated as the squared distance between the original and the reconstructed point), as well as the MSE in both sets of course.

Things I've tried:

• get the ratio of MSEs (>1 means the second distribution is "anomalous", but how much? what threshold can I set?)
• get the 95% percentile of both the error sets and get the ratio
• count the points beyond the 95% percentile of both the error sets and get the ratio

but none of these seems a good stable indicator.

What would you recommend?

Answer 1

I had some observations in a very similar setting:

• The error distribution on the training data is misleading since your training error distribution is not identical to test error distribution, due to inevitable over-fitting. Then, comparing training error distribution with future data is unjust. A better approach is to keep a portion of the training set out of the training session and using it exclusively for revealing the error distribution. You should note that, this portion of the data should be anomaly-free.

• How the two distribution look alike might not be a good idea in an anomaly detection setting since we are generally interested in point anomalies, e.g. anomalies of single samples. Thus, it is better to benchmark this single sample's MSE against a your error distribution, rather than comparing distributions.

• You can determine your threshold value through either a supervised or an unsupervised scheme. Supervised approach is simple. You can tune your threshold value as a hyper-parameter based on its success at detecting labelled anomalies. The key point here is not to use test data for this. Hyper-parameter tuning on the test data will lead to over-fitting. So, you should crop some validation data. If you do not possess any labelled data, then you should consult some statistical measures at the risk of losing some detection capability. You can simply set a threshold based on: 95% percentile or $Median+1.5*IQR$ (IQR: Interquartile range). You can also use z-scores, but do not forget that z-score's connection to the the percentiles depends on the shape of the distribution. In case of a Gaussian distribution, you can safely apply z-scores with a threshold of $z=3.5$. As another option, you can give a chance to Chebyshev’s Theorem (link) which applies to all kinds of distributions. It outputs an interval where $k\%$ of the data is guaranteed to sit on.