What are the impacts of different learning rates on this model and why does it keep overfitting?

Question

I am training a custom CNN model with 10 Layers based on this paper: https://journals.sagepub.com/doi/full/10.1177/1558925019897396

I have two classes (defect integrated circuit & non defect integrated circuit) with around 8.000 images after Data Augmentation for each class. I wanted to train this model from Scratch and used the Hyperparameters from the paper:

parameter	value
Learning Rate	10^-4
Epochs	2000
Val Split	0,25
Batch Size	64
Optimizer	Adam

My architecture is as follows:

input image is 384x384x3

conv 1-1 with 64 filters

max pool

conv 2-1 with 64 filters

max pool

conv 3-1 with 128 filters

conv 3-2 with 128 filters

max pool

conv 4-1 with 256 filters

conv 4-2 with 256 filters

max pool

conv 5-1 with 64 filters

max pool

dense with 512 units

dropout with 0.5 ratio

dense with 2 units

softmax

After training this model I see that the model is likely overfitting due to the validation loss stagnating:

Until Epoch 100 the validation loss decreases and it seems to be a good model but after that as you see the validation loss gets out of hand. After reducing the Learning rate to 10^-5 because it seemed that the weights are jumping too much so the model can't find a local minimum to stay in. But the result was this:

Now the validation loss doesn't only seem to be instable but it also increases over time. I also tried a learning rate of 10^-6, but it also seems not to be good:

So I tried with a learning rate of 10^-3 but that was clearly the worst model:

Now I know I could schedule the learning rate and start to decrease it over time, but my professor wants me to train the model with a "static" learning rate. How would you explain these training results? I thought decreasing the learnig rate would help the model but it clearly doesn't. Does that mean that the overfitting doesn't have to do with the learning rate (I know there are many factors to it) but maybe with the complexity of the architecture? Are there too many layers or why does it keep overfitting? Or would you just stick to a learning rate of 10^-4 and stop training around 100 epochs. Because this is what I get when I stop then:

The thing is 80 epochs are very low for a model that is trained from scratch or not?

Answer 1

Theoretically rigorous understandings of deep neural networks are still a wide-open area of research, though a multitude of statistical heuristics are in common practice. In this light, I'll try to go through your questions and offer some empirical suggestions.

1. What are the impacts of learning rate on this model?

At a high level, iterative optimization techniques, i.e. gradient descent or GD (Adam, the one that you use, can be thought of as a version of GD with some enhancements), calculate a "descent direction," and then update the model parameters in that descent direction by an amount given by the step size.

In the deep learning setting, the best learning rates are often found using hyperparameter search -- i.e. trying many different values and selecting the model with the best validation performance. This is what you are doing. In general, I've seen that these heuristics (which are actually provable in simpler regimes than deep learning) hold fairly well even in the deep learning regime:

a. If you set your learning rate too low, your model will converge very slowly.

b. If you set your learning rate too high, your model's convergence will be unstable; training loss may bounce around, or even get stuck at a suboptimal level (local minima).

I see this in your graphs: at learning rate $10^{-4}$, I actually think the model is performing quite well from an optimization standpoint; however, there is a small gap in the training and validation performance. Whether or not this is acceptable requires task-specific knowledge. When you bring down the learning rate by a factor of 10, the model learns slower: it's as if you've zoomed in on the left half of the previous graph -- which is very nice expected behavior.

In the regime with learning rate $10^{-3}$, the model pretty much fails to learn, which is potentially attributable to having overly large parameter updates that entirely "skip over" the region of "good" parameter values.

Even though you mention your PI wants you to use a constant learning rate, it is now common practice to use learning rate schedules, where the learning rate changes (usually decreases, excepting the usage of "warmup" -- beginner-friendly guide here) during training.

2. Why is it overfitting?

TL;DR I think you are overtraining a little.

You have a 10-layer CNN; this is very likely powerful enough to memorize the entire training dataset of 384x384x3 images. In fact, we see this in some of your graphs, where training accuracy reaches 1. I do indeed observe a small gap between training and validation performance; whether this is of concern depends on the task (i.e. is the validation performance satisfactory to an expert).

Decreasing the learning rate doesn't necessarily help overfitting; I could see an argument that in certain regimes, it would slow down the onset of overfitting, but the statistical problem remains. Some techniques that empirically help overfitting:

1. Dropout. You do this already in your final layers.
2. Regularization. The most common is weight decay regularization; it's built into most ML frameworks.
3. Changing the model architecture. In general, models with more parameters tend to overfit more readily, so removing layers or decreasing the layer sizes can help.
4. Early stopping. You do this in the final graph. This is very common practice; you'd set a rule along the lines of "if the validation loss fails to improve for N epochs, end training."

A tiny point of correction: you claim that we can see overfitting due to validation loss stagnating. Actually, overfitting is characterized by a large gap between training accuracy (usually 1 or very close to 1 in the overfitted remige) and validation accuracy. Here's a graph from Google Images that shows this:

Validation loss stagnation is, however, a common symptom of overfitting.