Reference to learn how to interpret learning curves of deep convolutional neural networks

Question

Question

What are good reference articles/blogs/tutorials to learn how to intepret learning curves for deep convolutional neural networks?

Background I am trying to apply convolutional neural networks (CNN) for vessel segmentation (specifically to determine whether or not the center pixel of an image patch is on a vessel) using caffe.

I have about 225000 training images (~50% positive) and 225000 (~50% positive) testing/validation images.

My input images are of size 65 x 65. I have four convolutional layers (48x6x6, 48x5x5, 48x4x4, 48x2x2) each followed by 2x2 max-pooling layers , one fully connected layer of 50 neurons, and a final scoring layer with 2 neurons. My training batch size is 256 and my testing batch size is 100.

I am using a stochastic gradient descent optimizer (SGD) and an inverse decay learning rate policy. Below are my caffe solver parameters:

• type: "SGD"
• base_lr: 0.01
• lr_policy: "inv"
• gamma: 0.1
• power: 0.75
• momentum: 0.9
• weight_decay: 0.0005

Below is the learning curve i am getting:

I am using the cross entropy classification loss or multinomial logistic loss (see here).

I would like to hear how people interpret this learning curve and what parameters they would change to try and improve the test accuracy.

• The training loss is decreasing but decreasing very slowly. Could it mean that my learning rate is low?

• On the contrary i see that the test loss decreases quickly at first and then slows down. Could this mean that my learning rate was high and it got stuck in the local minimum?

• Also the test accuracy has stabilized and stopped increasing too soon. Could this mean that i have to try and increase my model capacity or decrease my regularization?

In general, what would help me and also probably others is if someone could point out a reference article/book/blog-post that delves deeply into the interpretation of such learning curves with many example cases.

I found this blog post which was very helpful but there is not much about the interpretation of learning curves (atleast not to my satisfaction).

Answer 1

2 things:

1. You should probably switch your 50/50 train/validation repartition to something like 80% training and 20% validation. In most cases it will improve the classifier performance overall (more training data = better performance)
2. If you have never heard about "early-stopping" you should look it up, it's an important concept in the neural network domain : https://en.wikipedia.org/wiki/Early_stopping . To summarize, the idea behind early-stopping is to stop the training once the validation loss starts plateauing. Indeed, when this happens it almost always mean you are starting to overfit your classifier. The training loss value in itself is not something you should trust, because it will continue to decrease even when you are overfitting your classifier.

I hope I was clear enough, good luck in your work :)

Answer 2

I know its an ancient question but i guess if anyone else bumps across it looking for something similar...

I would say the barely decreasing loss value suggests using a lower learning rate? But i guess that's been mentioned in the link you included in your question so you probably considered that.

I would also try to change the filters you use in your model. From most of the stuff I've seen online usually they use 3x3 filters for all layers, the ones that use varying filter sizes also tends to go from 3x3 to 4x4 etc. I guess intuitively this allows you to extract more finer details in the initial layers and build up upon them?

In terms of resources this also seems interesting although also relatively limited: https://machinelearningmastery.com/diagnose-overfitting-underfitting-lstm-models/

I was also hoping someone who comes across this could check out the problem I have which is sort of similar in nature. I've tried changing a number of parameters but there is a large gap between training and validation loss which closes in suddenly after a few epochs.

How do I interpret my validation and training loss curve if there is a large difference between the two which closes in sharply

I would also find it EXTREMELY useful if someone could put together a reference suggesting practical suggestions of how to diagnose and address common issues in models / parameters from loss plots. Maybe even for these plots

http://lossfunctions.tumblr.com/

Answer 3

Here are some suggestions:

1. Training error and test error are too close (your system is unable to overfit on your training data), this means that your model is too simple. Solution: more layers or more neurons per layer.

2. Decrease gamma, e.g., 0.01. If the curve still reach plateau early, you can try a larger learning rate, e.g., 0.1.

3. Try Adam instead of SGD. The convergence of Adam is usually faster than SGD.