# Validation loss increases while Training loss decrease

I am training a model and the accuracy increases in both the training and validation sets. I am using a pre-trained model as my dataset is very small. I am not sure why the loss increases in the finetuning process for the validation: while when training from scratch, the loss decreases similar to the training:

I add the accuracy plots as well here: Fine tuning accuracy:

Training from scratch accuracy:

The model used in the pretraining did not have all the classes/nor exact patterns in the training set. Does that explain why finetuning did not enhance the accuracy and that training from scratch has a little bit enhancement compared to finetuning?

Extra Information:

I am using C3D model, which first divides one video into several "stacks" where one stack is a part of the video composed of 16 frames. I am trying to learn actions from videos.

The C3D model consists of 5 convolutional layers and 3 fully connected layers: https://arxiv.org/abs/1412.0767

Pretraining dataset: 11 classes, with 6646 videos divided into 94069 stacks Training dataset: 18 classes (with 11 "almost similar" classes to the pretraining), and 657 videos divided into 6377 stacks

In the fine tuning, I do not freeze any layers as the videos in the training are in different places compared to the videos in the dataset used for the pretraining, and are visually different than the pretraining videos. I am trying next to train the model with few neurons in the fully connected layer.

Learning rate starts with lr = 0.005 and is decreased after step 4, 8, 12 by 10, 100, 1000 respectively in both the pretraining and the fine-tuning phases

• Looks like you are overfitting the pre-trained model during the fine tuning. How many images do you have? Do you only train a fully connected layer (they are the one with most parameters)? Also, in my experience, and I think it is common practice that you'd want a pretty small learning rate when fine tuning using a pretrained model. As your validation error shoots up and training goes down, it may be that the learning rate is too large.
– Tom
Mar 3, 2019 at 13:24
• I am using C3D model which is trained on videos rather than images, I have added the required information in the question, thanks for pointing to the missing information. I am trying next to use a lighter model, with two fully connected layer instead of 3 and to use 512 neurons in the first, while the other layer contains the number of classes (dropped in the finetuning) Mar 4, 2019 at 10:16
• Looks like pre-trained model is already better than what you get by training from scratch. It would be useful to see the confusion matrices in validation at the beginning and end of training for each version Mar 4, 2019 at 11:11
• Why do u mention that the pre-trained model is better? The accuracy achieved by the training from scratch is better than the accuracy with finetuning. Mar 4, 2019 at 18:32

The reason you don't see this behaviour of validation loss decreasing after $$n$$ epochs when training from scratch is likely an artefact from the optimization you have used. The training rate has decreased over time so any effects of overfitting are mitigated when training from scratch. When fine-tuning the pre-trained model the optimizer starts right at the beginning of your training rate schedule, so starts out with a high training rate causing your loss to decrease rapidly as it overfits the training data and conversely the validation loss rapidly increases.