I have a simple question which I cannot find a straight answer to. I am training a neural network to classify some medical images. I initially focused on validation accuracy after each epoch (to determine how the network was generalising) and then after that, test accuracy on an unseen dataset.
But I see that validation loss is also important - and sometimes my validation loss drops even thought for that epoch, the validation accuracy also goes down slightly. Ultimately test accuracy is going to be the gold standard I guess, but thats no use for guiding training and adjusting parameters. Obviously if the the validation loss starts to go up and validation accuracy starts to drop it indicates overfitting. But what about when both validation loss and validation accuracy drop after an epoch?