5
$\begingroup$

I have a doubt about the computation of the loss and accuracy for the training / validation / test.

I split my training set into batches, and I'm training my network with them for N epochs. My idea is training the network with the k-th batch and than computing the loss using the entire training set, the final loss for the current epoch is the last loss computed on the entire training set after the last batch, which is not computationally efficient. Other people sum the loss of each batch and divide by the number of batches analyzed for getting the loss of the current epoch. Is there a difference between these two methods? I think the answer is yes, but does this difference significantly impact in the loss computation?

For the accuracy some people for each batch count the number of corrected classified sample with respect to the size of the batch and keep summing these values for all the batches, but I think that this could be not accurate because during the first batches the network have different weights and they are changing during the training phase. Should the accuracy be computed on the entire training after the last analyzed batch?, where the weights are fixed and they will not change anymore for the current epoch.

For the validation and the test set instead in my opinion is correct to computing the loss and the accuracy on the entire sets (or doing the same with batches if the set is too large) as the weights are not changing in this phase.

$\endgroup$
3
  • 1
    $\begingroup$ are you saying that you will recompute de loss for the entire training set after doing batches? $\endgroup$ Commented Nov 15, 2019 at 0:27
  • $\begingroup$ It is a possible version of my question. Because the weights are changing for every batch so I compute the loss only once after the batches on the entire training set. I don't understand why I have to sum all the losses and compute the average loss, i Think that it can work but the loss is distorted. $\endgroup$
    – FraMan
    Commented Nov 15, 2019 at 8:29
  • $\begingroup$ You should keep in mind that the actual metric that matters is the test set, batching is normally use when you have data that is bigger for your system, if you retrain for getting the loss as a whole basically you will be overfitting your dataset. You always want to consider your test and validation metrics. $\endgroup$ Commented Nov 15, 2019 at 18:33

3 Answers 3

3
$\begingroup$

To unbiasedly estimate a model's training loss at the end of an epoch, do exactly what you do to estimate its validation loss at the end of an epoch: set the model in evaluation mode (to disable training-only computations like dropout), and apply the model to the sample. Not all of the training sample has to be used here; a random subsample should do just fine.

This procedure is less biased than the typical procedure you mentioned:

sum the [training] loss of each batch and divide by the number of batches analyzed for getting the loss of the current epoch

The typical procedure saves time because there are no extra model calls; it just sums what was already computed. But it's a biased estimator of training loss because:

  1. See Sycorax's answer
  2. Training-only computations were applied to get the loss for each training batch, since (presumably) this loss is recycled from the forward pass during backpropagation. Applying dropout, for example, causes loss to be overestimated. There are other training-only computations, e.g., feeding the model back true rather than predicted sequence elements as in a "teacher forcing" architecture for language translation (e.g., Figure 10.6 here1), which cause training loss to be underestimated.

I personally prefer to compute an unbiased estimate of training loss and error because it's insightful to see how training and validation loss and error compare between different models. One can iterate a model more easily by understanding how much certain interventions affect its bias and variance.

The typical procedure is fine if your only goal is to sanity check that optimization is working, i.e., training loss consistently goes down. Training error doesn't have to be estimated at all to select the best performing model. Only validation error needs to be estimated.

References

  1. Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
$\endgroup$
2
$\begingroup$

Suppose that the training process is working & the loss is decreasing during an epoch. If this is the case, then we know that the average loss at the beginning of the epoch will be larger than the average loss at the end of the epoch. This means you have to make a choice:

  1. As a result, if you store all of the minibatch losses computed during the epoch, then compute the average, that average will be biased upwards, towards the loss value at the beginning of the epoch. The extent to which this matters depends on the magnitude of the difference between the loss at the beginning of the epoch and the loss at the end of the epoch. The closer together the beginning loss and end loss are, the smaller the bias will be.

  2. If you discard the minibatch losses computed during the epoch & recompute the loss for all samples at the end, then you're increasing the computational cost of each epoch because you're passing the training data to the model twice.

You'll have to decide which one is the best fit for your needs. If you have a tight budget, (1) might make more sense. If you have a great need to precisely measure the training loss, then (2) might be better.

$\endgroup$
-1
$\begingroup$

People average batches because the data is distributed among those batches, you have to consider that taking the best (or worst) would be taking the best scenario (or worst), reusing data is a risk of overfitting and you dont wanna do that. And as I said in the comment always report the test-validation for whatever metric you using.

$\endgroup$
2
  • $\begingroup$ I think that you don't understand what I said. I wrote a different thing. My idea is training the network with the k-th batch and than computing the loss using the entire training set, which not mean that I retrain the network $\endgroup$
    – FraMan
    Commented Nov 15, 2019 at 19:31
  • $\begingroup$ Perhaps some code may help... $\endgroup$ Commented Nov 15, 2019 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.