1
$\begingroup$

I'm reading this paper, which trains an autoencoder to perform background subtraction on an image. The use SGD batch training with binary cross entropy loss function to train the autoencoder on D training examples ${x_1,...,x_D}$. The trained networks response to the training examples are ${y_1,...y_D}$. However there stopping criteria seems to using the mean squared error loss, not the binary cross entropy loss. They stop training the network when $\sum_{i=1}^{D} |x_i -y_i|^2 < \delta$.

This is kind of peculiar to me. I've never seen this kind of stopping criteria. Usually the stopping criterion is when the validation loss stops improving, but they don't even have a validation set. Is there any reason they chose this criteria?

$\endgroup$
1
$\begingroup$

I wasn't able to access the paper you refer to but perhaps a few of the following insights might help answer your question:

  • Both cross entropy and MSE are frequently used for the stopping criterion when training autoencoders (not sure why the authors refer to cross entropy when they use MSE though)
  • When training an auto-encoder, the goal is effectively to reproduce the output y from input x as close as possible, therefore the stopping criterion they use does appear to make sense
  • Depending on the context, it is not always strictly necessary to use a validation set when doing unsupervised training. The risk is of course that overfitting occurs and that the autoencoder doesn't perform well on unseen training data. If the authors don't present any other unseen training data to the autoencoder afterwards, there is no need for a validation set.
  • Another possibility is that if the autoencoder is subsequently used as a part of a larger model afterwards, one might also judge the performance of the autoencoder on the outcome of the entire algorithm. The disadvantage with this approach is that you don't know the impact of the autoencoder on the performance of the entire model for unseen training data.
$\endgroup$
  • $\begingroup$ Thanks! Here's a link to the pdf $\endgroup$ – mdornfe1 Mar 2 '17 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.