0
$\begingroup$

I have a doubt about the computation of the loss and accuracy for the training / validation / test.

I split my training set into batches, and I'm training my network with them for N epochs. My idea is training the network with the k-th batch and than computing the loss using the entire training set, the final loss for the current epoch is the last loss computed on the entire training set after the last batch, which is not computationally efficient. Other people sum the loss of each batch and divide by the number of batches analyzed for getting the loss of the current epoch. Is there a difference between these two methods? I think the answer is yes, but does this difference significantly impact in the loss computation?

For the accuracy some people for each batch count the number of corrected classified sample with respect to the size of the batch and keep summing these values for all the batches, but I think that this could be not accurate because during the first batches the network have different weights and they are changing during the training phase. Should the accuracy be computed on the entire training after the last analyzed batch?, where the weights are fixed and they will not change anymore for the current epoch.

For the validation and the test set instead in my opinion is correct to computing the loss and the accuracy on the entire sets (or doing the same with batches if the set is too large) as the weights are not changing in this phase.

$\endgroup$
3
  • $\begingroup$ are you saying that you will recompute de loss for the entire training set after doing batches? $\endgroup$ Nov 15 '19 at 0:27
  • $\begingroup$ It is a possible version of my question. Because the weights are changing for every batch so I compute the loss only once after the batches on the entire training set. I don't understand why I have to sum all the losses and compute the average loss, i Think that it can work but the loss is distorted. $\endgroup$
    – FraMan
    Nov 15 '19 at 8:29
  • $\begingroup$ You should keep in mind that the actual metric that matters is the test set, batching is normally use when you have data that is bigger for your system, if you retrain for getting the loss as a whole basically you will be overfitting your dataset. You always want to consider your test and validation metrics. $\endgroup$ Nov 15 '19 at 18:33
0
$\begingroup$

People average batches because the data is distributed among those batches, you have to consider that taking the best (or worst) would be taking the best scenario (or worst), reusing data is a risk of overfitting and you dont wanna do that. And as I said in the comment always report the test-validation for whatever metric you using.

$\endgroup$
2
  • $\begingroup$ I think that you don't understand what I said. I wrote a different thing. My idea is training the network with the k-th batch and than computing the loss using the entire training set, which not mean that I retrain the network $\endgroup$
    – FraMan
    Nov 15 '19 at 19:31
  • $\begingroup$ Perhaps some code may help... $\endgroup$ Nov 15 '19 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.