0
$\begingroup$

This is my first question here, so forgive me if I'm providing too little or too much detail.

I am training a CNN for image classification of fog conditions on camera images. I have three classes: No fog, light fog and dense fog.

To train this model I use PyTorch, where I finetune a pre-trained Resnet-18 model that was trained on ImageNet data on my data. Specifically, I've removed the last Fully Connected layer and replaced it to train for a 3-class classification problem. Learning rate = 0.001.

The data was randomly split into 80% train data and 20% test data. Furthermore, there is a large class imbalance in my data, meaning that there are 152.333 images with no fog, 1187 images with light fog and 604 images with dense fog. To try and counteract this problem I have used PyTorch's WeightedRandomSampler, which makes sure that there is class balance in every training batch. I've checked this, and an equal amount of the three classes are shown in every batch.

Now comes my actual 'problem': The validation loss is lower than the training loss at the very first epoch and continues to stay lower. Also, this validation loss (and accuracy) seems to randomly fluctuate but always stays high. Here are the loss curves of the model training:

Loss curves

I cannot figure out what might be causing this strange behaviour. I thought that maybe the model was predicting most of the validation images to be the majority class, but this confusion matrix shows that it is actually not doing this:

enter image description here

0 = no fog 1 = light fog 2 = dense fog

The images were taken by 8 cameras that are static and thus shoot images of the same scenery all the time. Randomcrop and RandomHorizontalFlip are used for the training set.

Does anybody have any clue as to what might be causing this strange behaviour of the validation loss, or is this maybe something that is actually O.K. when training a CNN for image classification?

$\endgroup$
1
$\begingroup$

You need to use the same loss for training and validation if you want to compare them. Your training loss is taking into account sample weights (generated by the WeightedRandomSampler) in order to compensate the class inbalance: misclassification of minority class samples is penalized much more than misclassification of majority class samples.

In the validation loss, you are not doing any correction like this: all classes are penalized equally. From the confusion matrix it is obvious that the validation set is very inbalanced, therefore a small change on the performance on the class 0 results in a large change in the validation loss curve (simply because it gets multiplied by the number of samples of class 0).

The solution is to use weighted loss function also for validation and assign a weight $1/N_k$ to each sample, where $N_k$ is the number of samples of class $k$ in the validation set.

$\endgroup$
  • $\begingroup$ Thanks for your answer Jan. But if I'm correct, the WeightedRandomSampler only has the effect that the distribution of the classes is approximately uniform for every batch right? It does not explicitly weigh loss at training time and thus at validation time does not need reweighing. Please correct me if I'm wrong on this point. However, I will certainly try the weighted loss approach and see if it gives 'better' results. Shouldn't I use the inverse of the class frequencies for this weighted loss? $\endgroup$ – Tim Apr 20 '18 at 15:24
  • $\begingroup$ I don't know the details of the WeightedRandomSamples implementation, it is well possible that instead of generating weights it simply alters the training distribution; the effect is equal (all classes participate on the batch loss by same amount). You can use the same approach for the validation: instead of using weights, just alter the validation set distribution by replicating samples in minor classes, or, even better - less redundant computation, weighting losses generated by each sample by 1/N_k $\endgroup$ – Jan Kukacka Apr 21 '18 at 10:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.