I'm building an autoencoder to identify anomalies on numerical data. The input features have different scales (i.e. some take values from 0 to 5, while others can be much much higher) and most of them are positively skewed. The features are too many to look manually and transform them to follow a normal distribution. The long-tails of the features are where I'm expecting to find anomalies.

I started by firstly scaling the data (zero mean, unit variance) and then applying a min-max scaling. Based on the reconstruction error of the features, it seems that most anomalies refer to features with smaller skewness. This is reasonable (given the min-max scaling) since the features with long tails are compressed within a small range and only a few values are close to '1'. As a result, those features' reconstruction errors are usually small.

My sense is that I should not apply the min-max scaling in order to be able to compare the reconstruction errors of features with different scales. So, here are my questions:

  1. Is it acceptable for NNs to accept inputs outside the range [0, 1]? The scaled data can take values up to 120 with the vast majority being within the [-1, 0] range. The output activation function is 'linear'.
  2. Should I possibly apply any other transformation like (X-median(X))/IQR(X)?
  3. Should I blindly (given that it's not possible to review them all manually) apply any function on the input features to make them more 'normal' (like sqrt, log, box-cox)?
  4. Given all these possibilities, is there any way to compare the results of these models since the reconstruction errors cannot be directly compared?
  • $\begingroup$ 1) how many features are we talking about? 2) time series data or cross-sectional data? 3) please add some description of your actual problem. Are you looking at credit card fraud? Equipment failure? Something else? What’s the average probability of an anomaly in your data? $\endgroup$ – DeltaIV Jul 16 '18 at 13:48
  • $\begingroup$ The data has about 30 features but I'm going to build a separate model for each client, so that leads to a few hundreds of models. The data has not time-ordering and I'm working on the security-related use cases. There's no pre-defined anomaly percentage but it should be very low (e.g. below 1%). I have worked on some automated technique that estimates this. $\endgroup$ – Stergios Jul 16 '18 at 15:00
  • $\begingroup$ With 30 features and only 1% anomalies for individual, building separate models will require a lot of data. Deep Learning is notoriously data-hungry. How many data do you have for customer? $\endgroup$ – DeltaIV Jul 17 '18 at 9:29
  • $\begingroup$ I have enough data. My main concerns regard the data pre-processing before feeding them to the autoencoder. $\endgroup$ – Stergios Jul 17 '18 at 10:48
  • $\begingroup$ @Stergios any updates on this? I see that sometimes not normalizing/standardizing the cross-sectional data would be better when it comes to anomaly detection using autoencoders, how was your experience? also this paper does a study on how sensitive anomaly detection models are to the normalization step. monash.edu/business/ebs/research/publications/ebs/wp16-2018.pdf $\endgroup$ – Soyol Aug 4 '20 at 17:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.