There is plenty of empirical evidence that deep enough neural networks can memorize random labels on huge datasets (Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals, "Understanding deep learning requires rethinking generalization"). Thus in principle by getting a big enough NN we can always reduce the training error to extremely small values, limited in practice by numerical accuracy, no matter how meaningless the task.
Things are quite different for the generalization error. We cannot be sure that for each learning problem, there exists a learnable NN model which can produce a generalization error as low as desired. For this reason the first step is to
1. Set your expectations correctly
Find a reputable reference which tells you that there exists an architecture which can reach the generalization error you're looking for, on your data set or on the most similar one for which you can find references. For example, look here
What are the current state-of-the-art convolutional neural networks?
to find current (at the time of the answers) SOTA (State Of The Art) performance for CNNs on various tasks. It's a good idea to try to reproduce such results on these reference data sets, before you train on your own data set, as a test that all your infrastructure is properly in place.
2. Make sure your training procedure is flawless
All the checks described in the answers to question
What should I do when my neural network doesn't learn?
to make sure that your training procedure is ok, are a prerequisite for successful reduction of the generalisation error (if your NN is not learning, it cannot learn to generalise). These checks include, among the other stuff:
- unit tests
- dataset checks (have a look at a few random input/label samples for both the training set and test set and check that the labels are correct; check width and size of input images; shuffle samples in training/test set and see if it affects results; etc.)
- randomisation tests
- standardize your preprocessing and package versions
- keep a logbook of numerical experiments
3. Try to get superconvergence
“Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates” by Leslie N. Smith and Nicholay Topin shows that in some cases the combination of large learning rates with the cyclical learning rate method of Leslie N. Smith acts as a regulariser, accelerating convergence by an order of magnitude and reducing the need for extensive regularisation. Thus this is a good thing to try before
4. Setting your regularisation to the MAXXX
Regularisation often increases training time (bad), increases the training error and reduces the generalisation error (good), but too much regularisation can actually increase both errors (underfitting). For this reason, and because of the increase in training time, it’s often better to introduce the various regularisation techniques one at a time, after you successfully managed to overfit the training set. Note that regularisation by itself doesn’t necessarily imply your generalisation error will get smaller: the model must have a large enough capacity to achieve good generalisation properties. This often means that you need a sufficiently deep network, before you can see the benefits of regularisation.
The oldest regularisation methods are probably early stopping and weight decay. Some of the others:
- reduce batch size: smaller batch sizes are usually associated with smaller generalisation error, so this is something to try. However, note that some dispute the usefulness of minibatches: in my experience, they help (as long as you don’t have to use crazy small sizes such as $m=16$), but Elad Hoffer, Itay Hubara, Daniel Soudry Train longer, generalize better: closing the generalization gap in large batch training of neural networks disagree. Note that if you use batch norm (see below), too small minibatches will be quite harmful.
- use SGD rather than adaptive optimisers: this has been already covered by @shimao, thus I only mention it for the sake of completeness
- use dropout: if you use LSTMs, use standard dropout only for input and output units of a LSTM layer. For the recurrent units (the gates) use recurrent dropout, as first shown by Yarin Gal in his Ph.D. thesis. However, if you use CNNs, dropout is used less frequently now. Instead, you tend to…
- ...use batch normalisation: the most recent CNN architectures eschew dropout in favour of batch normalisation. This could be just a fad, or it could be due to the fact that apparently dropout and batch normalisation don’t play nice together (Xiang Li, Shuo Chen, Xiaolin Hu, Jian Yang, Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift). Since batch norm is more effective than dropout when you have huge data sets, this could be a reason why dropout has fallen out of favour for CNN architectures. If you use batch normalisation, verify that the distribution of weights and biases for each layer looks approximately standard normal. For RNNs, implementing batch norm is complicated: weight normalisation (Tim Salimans, Diederik P. Kingma, Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks) is a viable alternative.
- use data augmentation: it also has a regularising effect.
5. Hyperparameter/architecture search
If nothing else helps, you will have to test multiple different hyperparameter settings (Bayesian Optimization may help here) or multiple different architectural changes (e.g. maybe in your GAN architecture and for the data set you're working on, batch norm only works in the generator, but when added to the discriminator too it makes things worse). Be sure to keep track of the results of these long and boring experiments in a well-ordered logbook.
PS for a GAN it doesn't make much sense to talk about a generalization error: the above example was meant only as an indication that there's still a lot of alchemy in Deep Learning, and things that you would expect to work fine, sometimes don't, or vice versa something which worked ok many times, suddenly craps out on you for a new data set.