7
$\begingroup$

I am working on anomaly detection using an autoencoder neural network with $1$ hidden layer. This is an unsupervised setting, as I do not have previous examples of anomalies. The input data has patterns but also varies a lot, hence, is partly stochastic in nature.

For understanding purposes, I trained a (complete) autoencoder with dimensions input = $500$, hidden = $500$, output = $500$ and sigmoid functions in the hidden and output layer. My training data has dimension $X\in[0,1]^{5000\times500}$ (500 variables, 5000 samples). I used $3$ algorithms, with learning rate $0.01$, mini batch size $64$, and pretty much the standard algo-parameters in Keras/TensorFlow:

  1. standard stochastic gradient descent (SGD)
  2. advanced/extended SGD with Nesterov momentum $0.9$ and learning rate decay $10^{-8}$
  3. Adam optimizer ($\beta_1=0.9$, $\beta_2=0.999$, learning rate decay $0$)

The image below shows the corresponding error curves. In my case both keep decreasing (except for Adam) so I would say "keep training". On the other hand I know intuitively that I should not train so long because there must be some overfitting going on. So how do I know when to stop training, how would you interpret the result below? Would I be right, to just take Adam and use 250 epochs (even though it has a wide bias between training/validation sets)?

enter image description here

Improve this question
$\endgroup$
6
  • $\begingroup$ It seems like this question could be answered by (1) positing a definition of overfitting and (2) examining whether or not you observe phenomena which meet that definition. So, what's your definition of overfitting? $\endgroup$
    – Sycorax ♦
    Commented Jan 11, 2019 at 15:25
  • $\begingroup$ In the context of the autoencoder, it would seem to me that overfitting would be equivalent to the model approximating an identity function, i.e. if the reconstruction error approaches $0$ and it starts to simply pass inputs to the output layer without forming a latent space and learning feature patterns. However, I am not sure if the ADAM algorithm result above would already constitute this "approximation" due to its low error close to $0$, or if such a low error is still reasonable (my inputs take values in $[0,1]$). $\endgroup$
    – robot_2077198
    Commented Jan 11, 2019 at 15:41
  • 6
    $\begingroup$ For an autoencoder, approximating the identity function is the entire point! Indeed, this is exactly the objective function. What you don't want is for the autoencoder to "learn nothing" and actually be the identity function. The tricks to prevent the AE from learning nothing are straightforward: use a bottleneck, regularize some or all parts of the encoder and decoder, learn from noisy inputs, etc. If your AE is obtaining 0 training error and you're learning through a bottleneck or similar, your model has learned how to reconstruct the input from limited information: a representation! $\endgroup$
    – Sycorax ♦
    Commented Jan 11, 2019 at 15:50
  • $\begingroup$ Thanks a lot for this very helpful response. Am I correct in understanding that we want to construct a "constrained" AE (be it via bottleneck, noisy inputs, or regularization) and then we would like to reduce the reconstruction error as much as possible (ideally $0$)? So as long as we enforce a restriction, which forces the AE to form representations, we really want the error to go to ideally $0$? $\endgroup$
    – robot_2077198
    Commented Jan 11, 2019 at 16:33
  • 1
    $\begingroup$ Your hidden layer should be smaller than your input. From there as long as your validation error doesn't start to increase you should be fine. $\endgroup$
    – Frobot
    Commented Jun 6, 2019 at 4:38

2 Answers 2

Reset to default
2
$\begingroup$

As validation tells about generalization of the algorithm. And from your graph, ADAM is working very good, with a biased response. But for sure, there is no overfitting sign in there.

For biasing check, you can try out k-fold method, and check the response of algorithm for each fold. Then you can find, whether this is irreducible error or something else.

Improve this answer
$\endgroup$
1
$\begingroup$

@Sycorax answer is what comes the closest to the answer.

Usually, overfitting is described as the model training error going down while validation error goes up, which means the model is learning patterns that don't generalize beyond the training set.

In the case of an autoencoder, you're training the model to reproduce the input. At an extreme your model could simply be output = input and both validation and training loss would be 0.

But that's not what you want from an autoencoder for anomaly detection. You want the model to learn an abstract representation of what the input should look like, so the model develop the ability to generate instances on it's own when presented with an input.

The distance or error between what the model "think it should look like" and the input is what tells us if the input is "abnormal" or not.

Best way to assess it would be to present the model with valid and invalid instances and monitor the reconstruction error. If it tends to be the same on both set your model is most probably learning the identity function and needs to be modified.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.