In addition to Franck's answer about practicalities, and David's answer about looking at small subgroups – both of which are important points – there are in fact some theoretical reasons to prefer sampling without replacement. The reason is perhaps related to David's point (which is essentially the coupon collector's problem).
In 2009, Léon Bottou compared the convergence performance on a particular text classification problem ($n = 781,265$).
Bottou (2009). Curiously Fast Convergence of some
Stochastic Gradient Descent Algorithms. Proceedings of the
symposium on learning and data science. (author's pdf)
He trained a support vector machine via SGD with three approaches:
- Random: draw random samples from the full dataset at each iteration.
- Cycle: shuffle the dataset before beginning the learning process, then walk over it sequentially, so that in each epoch you see the examples in the same order.
- Shuffle: reshuffle the dataset before each epoch, so that each epoch goes in a different order.
He empirically examined the convergence $\mathbb E[ C(\theta_t) - \min_\theta C(\theta) ]$, where $C$ is the cost function, $\theta_t$ the parameters at step $t$ of optimization, and the expectation is over the shuffling of assigned batches.
- For Random, convergence was approximately on the order of $t^{-1}$ (as expected by existing theory at that point).
- Cycle obtained convergence on the order of $t^{-\alpha}$ (with $\alpha > 1$ but varying depending on the permutation, for example $\alpha \approx 1.8$ for his Figure 1).
- Shuffle was more chaotic, but the best-fit line gave $t^{-2}$, much faster than Random.
This is his Figure 1 illustrating that:

This was later theoretically confirmed by the paper:
Gürbüzbalaban, Ozdaglar, and Parrilo (2015). Why Random Reshuffling Beats Stochastic Gradient Descent. arXiv:1510.08560. (video of invited talk at NIPS 2015)
Their proof only applies to the case where the loss function is strongly convex, i.e. not to neural networks. It's reasonable to expect, though, that similar reasoning might apply to the neural network case (which is much harder to analyze).