4
$\begingroup$

I train a neural network on some data using minibatches. I evaluate it as follows: for the training set, I check the performance only on the current minibatch. For the Dev (or test) set, I check the performance on the entire set.

I get two phenomena (image attached, each point is the evaluation after a minibatch finishes, red arrows denote start of new epochs):

  1. For the training error, at the start of each new epoch I get a serious 'bump' up in performance (i.e. overfitting). I assume this is because of my incomplete evaluation - I don't evaluate the entire training set each minibatch but only the current minibatch.

  2. For the Dev error - which is my real concern - I get some deterioration after immediately at the start of the second epoch (and the phenomena repeats itself on future epochs). While I know that the Dev performance should indeed diminish at some point, I don't understand why it happens exactly on the start of an epoch.

Is this reasonable (especially item 2.)? or do I have some bug in my code?

Thanks.

train-dev performance

$\endgroup$
4
$\begingroup$

Yes, this reduction is normal

When training any kind of machine learning algorithm, if you continue training, your algorithm overfits the training set and starts learning the details of the noise in the training set instead of utilizing the generalizable information. When it does this, your algorithm often loses it's generality and gets worse on other similar sets, specifically your dev set.

I have seen jumps like this myself when fitting an algorithm to my training set. It finds a pattern in the noise of the test set, but this pattern does not generalize to the test set.

There are a few ways to reduce overfitting:

  1. Cross-validation - Hold out a subset of your training data (most use ~10%) and then use that to compare your algorithm and find a good stopping point. This prevents you from potentially overfitting your dev set as well.
  2. Regularization - Add a penalty to your loss function for each NN node, so that it focuses on the most important connections.
  3. Dropout - Randomly drop various connections as your train each epoch. This forces the NN to reduce reliance on particular connection webs and makes each node more robust by itself since it might not be able to rely on other nodes that could potentially be dropped out of each epoch.
  4. Stop training at the point when the test error starts decreasing successively. That might be the point at which the algorithm has learned what it can and it is no longer beneficial to train with the training data that you have.

I hope this helps!

$\endgroup$
2
  • $\begingroup$ As I recall from previous experience, the data indeed overfits eventually, but the deterioration of the dev performance doesn't happen in such a pronounced - manner exactly at the beginning of some epoch. Do you think this phenomenon is overall reasonable and does not indicate some unexpected behaviour, just 'normal' overfitting? $\endgroup$
    – yoki
    Oct 15 '16 at 15:29
  • 1
    $\begingroup$ This is reasonable because a new epoch starts to re-use the data from the training set. That would be the most likely time for over-fitting to reduce its generalization power. That said, it is possible that there is an error with your setup, but that looks like very typical over-fitting for an epoch-based algorithm. One thing to consider is whether your train set is shuffled at the beginning of each epoch. If not, this means that it might be overfitting on some extreme values that do not match the rest of our data at the beginning of each set. It is hard to say without knowing secifics. $\endgroup$
    – Tony S
    Oct 16 '16 at 23:23
1
$\begingroup$

I believe this can be an artefact on how you calculate your metrics. If you are using an moving average to calculate a metric and reseting the moving average at epoch boundaries than it is reasonable to expect a jump in epoch boundaries.

For example if you accuracies are going from 0 to 50 in the first epoch you moving average at the end of your first epoch will be less than 50. Depending on the window size assuming you report at the whole batch level it will be 25. But at the start of the next epoch it will start from 50 hence the jump.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.