I am working on a binary classification problem for a while now, I am using Keras to build my CNN model. So far I have made some progress to improve the accuracy result by tuning some important hyperparameters such as learning rate, l2 regularization lambda and learning rate decay.
My training data accuracy increases with the number of epochs but I can't see any consistent result in validation accuracy.
Please see below screen shot for one of my training:
Can anyone shed some light on what could be wrong or what could be the reasons for this kind of behaviour? Any logical and insightful thoughts would be appreciated.
PS - I am using augmented data for training but my validation set consists only raw samples.