validation accuracy, recall and precision remains constant after 30th epoch

Question

I am using TensorFlow model EfficientNetB0 for transfer learning, but after a number of epochs the validation accuracy, -precision, and -recall remains constant. Is this something I should be worried about? Also, I have 158 test files, but when I count up the values in the confusion matrix I only get 144.

Here is the code and output:

import os
import tensorflow as tf
import matplotlib.pyplot as plt
from tensorflow.keras import layers
from google.colab import drive
from keras.preprocessing.image import ImageDataGenerator
from sklearn.metrics import accuracy_score, f1_score, precision_score, confusion_matrix


img_size = (224,224)
batch = 16

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
    train_dir,
    validation_split = 0.25,
    subset = 'training',
    seed=123,
    image_size= img_size,
    batch_size = batch,
    label_mode = 'binary'
)

val_ds = tf.keras.preprocessing.image_dataset_from_directory(
    train_dir,
    validation_split = 0.25,
    subset = 'validation',
    seed=123,
    image_size= img_size,
    batch_size = batch,
    label_mode ='binary'
)

test_ds = tf.keras.preprocessing.image_dataset_from_directory(
    test_dir,
    #seed=123,
    image_size= img_size,
    batch_size = batch,
    #label_mode = None
)

class_names = test_ds.class_names

AUTOTUNE = tf.data.AUTOTUNE

train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)
test_ds = test_ds.prefetch(buffer_size=AUTOTUNE)


preprocess_input = tf.keras.applications.efficientnet.preprocess_input

data_augmentation = tf.keras.Sequential([
  tf.keras.layers.experimental.preprocessing.RandomFlip('horizontal'),
  tf.keras.layers.experimental.preprocessing.RandomFlip('vertical'),
  tf.keras.layers.experimental.preprocessing.RandomRotation(0.2)
])

base_model = tf.keras.applications.EfficientNetB0(input_shape = (224, 224, 3),
                                                    include_top = False,
                                                    weights = 'imagenet')

base_model.trainable = False
global_average_layer = tf.keras.layers.GlobalAveragePooling2D()
prediction_layer = tf.keras.layers.Dense(1)

inputs = tf.keras.Input(shape=(224,224, 3))
x = data_augmentation(inputs)
x = preprocess_input(x)
x = base_model(x, training=False)
x = global_average_layer(x)
x = tf.keras.layers.Dropout(0.2)(x)
x = prediction_layer(x)
outputs = tf.nn.sigmoid(x)
model = tf.keras.Model(inputs, outputs)

base_learning_rate = 0.0001
model.compile(optimizer=tf.keras.optimizers.Adam(lr=base_learning_rate),
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),
              metrics=['accuracy','Recall', 'Precision', 'FalsePositives', 'TruePositives', 'FalseNegatives'])

epochs=100
history = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=epochs
)

loss, accuracy, recall, precision, fp, tp, fn = model.evaluate(test_ds)
print('Test accuracy :', accuracy)
print('Test recall :', recall)
print('Test precision :', precision)
print('False Positive :', fp)
print('True Positive :', tp)
print('False Negative :', fn)
tn = 144-fp-tp-fn
print('True Negative :', tn)

Output:

Answer 1

As long as you have real-world data, there will always be remaining variation that your models cannot account for. There may be two cases with exactly the same predictor settings, but one is class A, the other class B.

Therefore, your KPIs will not asymptotically approach 1, no matter how long you train. Instead, they will always plateau somewhere below 1.

Also: Do not use accuracy to evaluate a classifier: Why is accuracy not the best measure for assessing classification models? Is accuracy an improper scoring rule in a binary classification setting? Classification probability threshold The same problems apply to sensitivity and specificity, and indeed to all evaluation metrics that rely on hard classifications. Instead, use probabilistic classifications, and evaluate these using proper scoring rules.

Answer 2

The accuracy, precision, and recall hitting a plateau while the loss does not is reasonable behavior. Accuracy, precision, and recall all just depend on the side of some threshold (basically always $0.5$ as a software default) where the prediction falls. Thus, the training might reach a point where basically every instance is on the correct side of that threshold but still be able to improve the raw predictions, as opposed to the binned predictions (classifications).

Further, despite the issues with accuracy, precision, and recall, your values look quite high, so your model seems to be good at something despite the fact that the loss value might be able to be improved.

It might be worth continuing to train to improve the loss, even though the threshold-based metrics are nearing a plateau. That little bit of additional loss you squeeze out of the model should result in better probabilistic prediction (which are useful) and might even flip a few observations to the correct side of the threshold to yield better accuracy, precision, and recall.