When does my autoencoder start to overfit? I am working on anomaly detection using an autoencoder neural network with $1$ hidden layer. This is an unsupervised setting, as I do not have previous examples of anomalies. The input data has patterns but also varies a lot, hence, is partly stochastic in nature.
For understanding purposes, I trained a (complete) autoencoder with dimensions input = $500$, hidden = $500$, output = $500$ and sigmoid functions in the hidden and output layer. My training data has dimension $X\in[0,1]^{5000\times500}$ (500 variables, 5000 samples). I used $3$ algorithms, with learning rate $0.01$, mini batch size $64$, and pretty much the standard algo-parameters in Keras/TensorFlow:


*

*standard stochastic gradient descent (SGD)

*advanced/extended SGD with Nesterov momentum $0.9$ and learning rate decay $10^{-8}$

*Adam optimizer ($\beta_1=0.9$, $\beta_2=0.999$, learning rate decay $0$)


The image below shows the corresponding error curves. In my case both keep decreasing (except for Adam) so I would say "keep training". On the other hand I know intuitively that I should not train so long because there must be some overfitting going on. So how do I know when to stop training, how would you interpret the result below? Would I be right, to just take Adam and use 250 epochs (even though it has a wide bias between training/validation sets)?

 A: As validation tells about generalization of the algorithm. And from your graph, ADAM is working very good, with a biased response. But for sure, there is no overfitting sign in there.
For biasing check, you can try out k-fold method, and check the response of algorithm for each fold. Then you can find, whether this is irreducible error or something else.
A: @Sycorax answer is what comes the closest to the answer.
Usually, overfitting is described as the model training error going down while validation error goes up, which means the model is learning patterns that don't generalize beyond the training set.
In the case of an autoencoder, you're training the model to reproduce the input. At an extreme your model could simply be output = input and both validation and training loss would be 0.
But that's not what you want from an autoencoder for anomaly detection.
You want the model to learn an abstract representation of what the input should look like, so the model develop the ability to generate instances on it's own when presented with an input.
The distance or error between what the model "think it should look like" and the input is what tells us if the input is "abnormal" or not.
Best way to assess it would be to present the model with valid and invalid instances and monitor the reconstruction error. If it tends to be the same on both set your model is most probably learning the identity function and needs to be modified.
