1
$\begingroup$

I have a binary classification problem where I'm trying to classify whether a given cell is cancerous or not. For this I decided to play around with VGG16 pre-trained model and simply remove the last dense layer which is used to classify the 1000 classes required for the imagenet and instead make it 2 class dense with softmax as activation.

How the model looks: model summary

The problem:

tensorboard of two models

Looking at the tensorboard where I compare 2 models that I've trained I can't seem to make sense of the results. It seems that after 30 epochs of training my validation loss begins increasing, but validation accuracy stays static, what gives? surely as my predictions begin deviating, the accuracy should change because the deviation should cause enough change on the weights to cause softmax to begin missclassifying images. Looking at my tensorboard it seems not to be the case.

The only way this makes sense is that if my validation set contains only of images of the same class and the model is constantly just guessing the same thing and just by chance it's guessing the right class constantly, but I checked my folder structure and the images are categorised as following:

positive-negative
 |
 +--test:
 |   +--negative:77
 |   +--positive:77
 +--train:
 |   +--negative:154
 |   +--positive:154
 +--valid:
 |   +--negative:26
 |   +--positive:26

This is how my batches are setup:

train_batches = ImageDataGenerator(rescale=1./255).flow_from_directory(fp_train, target_size=(width,height), classes=["negative","positive"], batch_size=2)
test_batches = ImageDataGenerator(rescale=1./255).flow_from_directory(fp_test, target_size=(width,height), classes=["negative","positive"], batch_size=test_size)
valid_batches = ImageDataGenerator(rescale=1./255).flow_from_directory(fp_valid, target_size=(width,height), classes=["negative","positive"], batch_size=2)

And my fitting configuration:

VGG16_Sequential.fit_generator(train_batches, 
    steps_per_epoch=154, 
    validation_data=valid_batches, 
    validation_steps=26, 
    epochs=epochs,
    verbose=1,
    callbacks=[tf_board, cp_cb])

What could possibly explain these results? I checked the validation set images and they are out-of-sample from the training and I clearly have two distinct cateogories.

$\endgroup$
1
$\begingroup$

The loss function is not just a yes/no prediction, it takes into account how confident the model was about correct and wrong predictions. As you may realize, a sigmoid acitivation actually predicts something that is a lot like a probability (for a neural network it is usually not nicely calibrated like a proper predicted probability). In contrast, accuracy is just a proportion of correctly classified items. Correctly classified in the context simply means those cases, for which the predicted probability is above the chosen threshold for positive examples and below for negative examples; you are probably just using a threshold of 0.5.

So, if for the images your model gets wrong, it keeps being more and more confident about its wrong predictions (and/or less confident about the correct ones), your loss function can get worse without the accuracy notably changing.

$\endgroup$
  • $\begingroup$ The way I understand loss is that it's the distance between the prediction and the ground-truth label. I just don't understand how my prediction can remain the same when the loss which is the difference is always increasing. At some point my predictions should constantly be false for both of these classes because the threshold should be crossed and the model should begin predicting wrong class. $\endgroup$ – Vocaloidas Apr 2 at 13:31
  • $\begingroup$ The binary crossentropy loss is -log(predicted probability)/N for items in the positive class and -log(1-predicted prob)/N for items in the negative class. So, let's say you get some cases in the positive class wrong. If you initially predict a probability of 0.4 of them being in the positive class (i.e. you predict negative, because 0.4<0.5), then 0.1, then 0.01, then 0.001 and so on as training progresses, then the accuracy does not change change, but the loss increases for such an item from 0.92/N to 2.30/n to 4.61/N to 6.91/N. $\endgroup$ – Björn Apr 2 at 14:02
  • $\begingroup$ Ah, see I was thinking about it interms of MSE. So the issue with my model is that it guesses correctly, but as the training continues, it begins to guess at around the 0.5 threshold where there is a lot of uncertainty for both of these classes, correct? Thanks for the explanation, really appreciate it. $\endgroup$ – Vocaloidas Apr 2 at 14:56
  • $\begingroup$ I think it's more that it becomes more confident of wrong predictions, but you can just test that and see. $\endgroup$ – Björn Apr 2 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.