I am currently trying to classify clothes for my final project in school. My problem is that after I gathered more data, to counteract overfitting, the validation accuracy dropped from 60% to 45%. Below I explain in detail what I did. I use the following network layout:
I have five different clothing classes: T-Shirt, Pullover, Hoodie, Jeans and Shorts.
I first gathered data from Image-net.org. I had about 700 images per class. I then started training the network, resulting in the following graph:
Clearly there was overfitting happening so I gathered more data for the jeans and shorts:
While the overfitting was still there, it started much later. However, the validation accuracy also got worse.
Now the overfitting started to look much better. However, the validation accuracy got much worse! What am I doing wrong here? Is there just not enough training data or is it something else?