I've often seen neural networks trained by shuffling all training examples each epoch and then splitting examples into batches and processing these batches sequentially. This way, each example is seen exactly once in an epoch. The training then usually goes for some fixed number of epochs.

Another possibility is to simply sample random batches (with replacement) from all training examples for a set number of batches. I've sometimes chosen this route because I couldn't fit all examples into memory.

Why does everyone seem to prefer the first method? Is it only because each example is seen exactly once per epoch? Does it typically make a difference in final testing accuracy? Certain forms of noise have been shown to be good for NN training, so maybe the random sampling with replacement could even be beneficial?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.