# High accuracy on both training and validation but very low on test set

My CNN model has about 96~97% accuracy on both training and validation sets. But when submitting the test set it got only 24% accuracy. Here's my model:

def build_cnn_model():
classifier = Sequential()
classifier.add(Convolution2D(32, (3, 3), input_shape=(64, 64, 3), activation='relu'))

return classifier


The training set has about 40k images, valid set has about 10k images and test set is made of 5.5k images. Here's my implementation

train_datagen = ImageDataGenerator(rescale=1./255)
valid_datagen = ImageDataGenerator(rescale=1./255)
test_datagen = ImageDataGenerator(rescale=1./255)

training_set = train_datagen.flow_from_directory(
'datasets/training_set',
target_size=(64, 64),
batch_size=32,
seed=42,
class_mode='categorical')

valid_set = valid_datagen.flow_from_directory(
'datasets/valid_set/',
target_size=(64, 64),
batch_size=32,
seed=42,
class_mode='categorical')

test_set = test_datagen.flow_from_directory(
'original_data/',
classes=['test'],
target_size=(64, 64),
seed=42,
class_mode=None,
batch_size=1)

test_set.reset()

classifier = build_cnn_model()

classifier.fit_generator(
training_set,
epochs=10,
steps_per_epoch=1222,
validation_data=valid_set,
validation_steps=305)


Here we can see the behavior of the model during training and I noticed that validation accuracy is always higher than training accuracy. So why this is happening? Why so low accuracy on test set and why validation accuracy is higher than training accuracy? What are the possible solutions?

• Your test generator has class_mode=None, which differents from train/val. Is that intentional? – jonnor May 23 at 11:53