Neural Network - Learning accuracy drops heavily after a couple of epochs

Question

I designed a neural network to classify some images into 28 classes. Here are the parameters :

• Weight Decay : 0.005
• Momentum : 0.01
• Learning Rate : 0.001 and 0.005
• Learning Decay : 1
• Input : 100x100 images
• Output : 28 Classes
• Training Data : 1500 Images
• Test Data : 30 Images
• Batch Learning
• Using Python with PyBrain

The images are Memes and each class has nearly 50 images in it. images in each class are highly simialr (because they're memes). no matter how many different networks I ran, it ended up like this:

The network learns to 100% and for the test data (nearly 30 images) it can classify up to 85%... BUT after a few more epochs, it loses accuracy and becomes completely useless... I tested many Hyper Parameters (with 50 epochs) to choose these parameters in the end but I guess I didn't test them for more than 50 epochs...

Here are my questions:

1. If I stopped training the network when the network reached Maximum Training and Testing accuracy, would the network be useful or the real network shows itself after more epochs thus becoming useless ?

2. Why does something like this happen? why does it lose accuracy after 25 epochs? What am I doing wrong?

Thanks

Answer 1

Neural network optimization is often like traversing a narrow canyon: if you take a step too large in the wrong direction, you can end up bouncing off of the canyon floor up the side and get stuck well outside of the canyon.

I think that's what's happened here. Your network made a "misstep" and then got stuck. Two common causes for this are

1. gradient descent step size is too large;

2. gradients can be "too large" for some steps. Gradient clipping can help.