I am building some image classification model with reasonable size data (~3K) images in both training and validation set. However, I noticed the performance on validation set is not stable.
For example, here are outputs over 10 epochs (acc means accuracy binary classification on balanced data)
epoch [1]: training loss: 2.27 - acc: 0.50 - val_loss: 3.29 - val_acc: 0.49
epoch [2]: training loss: 1.55 - acc: 0.50 - val_loss: 0.92 - val_acc: 0.50
epoch [3]: training loss: 1.07 - acc: 0.51 - val_loss: 1.43 - val_acc: 0.53
epoch [4]: training loss: 0.87 - acc: 0.58 - val_loss: 1.85 - val_acc: 0.61
epoch [5]: training loss: 0.59 - acc: 0.72 - val_loss: 0.58 - val_acc: 0.61
epoch [6]: training loss: 0.52 - acc: 0.79 - val_loss: 2.30 - val_acc: 0.50
epoch [7]: training loss: 0.38 - acc: 0.85 - val_loss: 0.17 - val_acc: 0.86
epoch [8]: training loss: 0.32 - acc: 0.88 - val_loss: 1.52 - val_acc: 0.60
epoch [9]: training loss: 0.21 - acc: 0.91 - val_loss: 0.14 - val_acc: 0.88
epoch [10]: training loss: 0.34 - acc: 0.88 - val_loss: 2.81 - val_acc: 0.49
We can see that in training, it seems fine, but for epoch 6 and 8 validation loss was very high, and the final epoch 10, the validation loss got so high that the model become useless.
What could be the reason causing this? If it is overfitting on training data, why we are not seeing steady increase on validation loss?