I'm using a recurrent neural network for prediction. How should I avoid from overtraining? I've used gradient descent for parameter updates. The following figure shows the training and validation error for 500 epochs:
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ Can you give us a little bit more information about your setup. What type of recurrent net are you using? Are you doing any type of regularization? What's your train size/feature size/batch size etc. This will help us diagnose your problem. $\endgroup$– Armen AghajanyanDec 15, 2015 at 19:49
-
$\begingroup$ I've had good luck combining dropout and maxnorm on RNNs. YMMV. $\endgroup$– Sycorax ♦Feb 19, 2017 at 13:53
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Many familiar techniques for avoiding overtraining are entirely relevant to recurrent neural networks. Most obviously, regularisation (typically L1 regularisation, where a penalty is proportion to the sum of absolute values of all weights and L2 regularisation, where the penalty is proportional the sum of the squares of the weights). Another neural network specific technique that is of interest is dropout. There is some evidence that dropout needs to be modified a bit to work well on RNNs (see Zaremba, Sutskever and Vinyals, ICLR 2015), specifically that it is best to avoid dropout on the recurrent connections (the ones that link across time steps).
Early stopping is a form of regularisation. With this you need to store copies of the neural network (most simply, one for the network that has achieved the lowest validation set error so far). Then you simply keep this network that performed best out of sample. This is not a perfect method, but it is very simple and very fast, as you could do without cross-validation. I believe Geoffrey Hinton expressed the informed view that for the best results, you should use a network from slightly earlier in the training than the absolute low of test set error (because the actual low is partly due to good generalisation and partly due to random fluctuations in test set score which do not generalise).
For completeness, I'll mention that a final form of regularisation that has proven useful in tasks with limited noisy data is to construct multiple variants of the examples you have with forms of distortion that reflect knowledge you have. For example, if you were training an RNN to write music, you might expand a dataset of tunes by transposing each to 12 different keys. In a sense you could think of this as encouraging your RNN to respect an intrinsic symmetry.
See http://arxiv.org/abs/cs/0410015 and http://arxiv.org/abs/1312.4569
-
$\begingroup$ This answer is a good start, but I'd be more inclined to upvote it if you filled in the missing details -- for example, what paper are you referencing (who wrote it, when, what journal)? $\endgroup$– Sycorax ♦Mar 11, 2016 at 16:38