0
$\begingroup$

I'm training a CNN in keras with tensorflow backend with the following model architecture for a binary classification problem. I've divided approximately 41k images into training, validation and test sets in the ratio 70:25:5 giving 29k images in train set, 10k in validation and 2k in test set.

There is no class imbalance, there were approximately 20k samples in each of pos and neg classes.

model = Sequential()
model.add(Conv2D(32, (7, 7), padding = 'same', input_shape=input_shape))
model.add(Conv2D(32, (7, 7), padding = 'same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.5))

model.add(Conv2D(32, (7, 7), padding = 'same'))
model.add(Conv2D(32, (7, 7), padding = 'same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.6))

model.add(Conv2D(32, (7, 7), padding = 'same'))
model.add(Conv2D(32, (7, 7), padding = 'same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.6))

model.add(Conv2D(64, (7, 7), padding = 'same'))
model.add(Conv2D(64, (7, 7), padding = 'same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.6))

model.add(Conv2D(64, (7, 7), padding = 'same'))
model.add(Conv2D(64, (7, 7), padding = 'same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.6))

model.add(Conv2D(64, (7, 7), padding = 'same'))
model.add(Conv2D(64, (7, 7), padding = 'same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.6))

model.add(Conv2D(128, (7, 7), padding = 'same'))
model.add(Conv2D(128, (7, 7), padding = 'same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.6))

model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))

model.add(Dense(512))
model.add(Activation('relu'))

model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer=optimizers.Adam(lr=3e-5),
              metrics=['accuracy'])

checkpoint = ModelCheckpoint(filepath='checkpointORCA_adam-{epoch:02d}-{val_loss:.2f}.h5', monitor='val_loss', verbose=0, save_best_only=True)

reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5,
                              patience=20, min_lr=1e-8)

train_datagen = ImageDataGenerator(rescale=1. / 255,
        shear_range=0.2,
        zoom_range=0.2)

# this is the augmentation configuration we will use for testing:
# only rescaling
test_datagen = ImageDataGenerator(rescale=1. / 255)

# Change the batchsize according to your system RAM
train_batchsize = 32 # changed them to 64 and 128 respectively, but same 
                      results
val_batchsize = 32

train_generator = train_datagen.flow_from_directory(
    train_data_path,
    target_size=(img_width, img_height),
    batch_size=train_batchsize,
    class_mode='binary',
shuffle=True)

# train_generator.reset()
# validation_generator.reset()
validation_generator = test_datagen.flow_from_directory(
    validation_data_path,
    target_size=(img_width, img_height),
    batch_size=val_batchsize,
    class_mode='binary',
shuffle=False)

# validation_generator.reset()

history = model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size,
    callbacks=[checkpoint, reduce_lr])

These are the epochs for the training progress, where the validation accuracy flucatuates in a linear fashion. It first gets high and then low by nearly the same amount. What could be the reason for this?

I've checked out nearly every other answer for this, and my data is normalized, properly shuffled in the training set, lr is small and very well checked in within what other researchers in similar problem domain found success with.

Found 29124 images belonging to 2 classes.
Found 10401 images belonging to 2 classes.
Epoch 1/60
910/910 [==============================] - 530s 582ms/step - loss: 0.6105 - acc: 0.6161 - val_loss: 0.2298 - val_acc: 0.9548
Epoch 2/60
910/910 [==============================] - 520s 571ms/step - loss: 0.3590 - acc: 0.8480 - val_loss: 0.8340 - val_acc: 0.6604
Epoch 3/60
910/910 [==============================] - 520s 571ms/step - loss: 0.3160 - acc: 0.8695 - val_loss: 0.0983 - val_acc: 0.9558
Epoch 4/60
910/910 [==============================] - 528s 580ms/step - loss: 0.2925 - acc: 0.8830 - val_loss: 0.5063 - val_acc: 0.8385
Epoch 5/60
910/910 [==============================] - 529s 581ms/step - loss: 0.2718 - acc: 0.8895 - val_loss: 0.0541 - val_acc: 0.9745
Epoch 6/60
910/910 [==============================] - 530s 583ms/step - loss: 0.2523 - acc: 0.8982 - val_loss: 0.5849 - val_acc: 0.8060
Epoch 7/60
910/910 [==============================] - 528s 580ms/step - loss: 0.2368 - acc: 0.9076 - val_loss: 0.0682 - val_acc: 0.9695
Epoch 8/60
910/910 [==============================] - 529s 582ms/step - loss: 0.2168 - acc: 0.9160 - val_loss: 0.6503 - val_acc: 0.7660
Epoch 9/60
910/910 [==============================] - 527s 579ms/step - loss: 0.1996 - acc: 0.9213 - val_loss: 0.0339 - val_acc: 0.9850
Epoch 10/60
910/910 [==============================] - 529s 581ms/step - loss: 0.1896 - acc: 0.9258 - val_loss: 0.5710 - val_acc: 0.8033
Epoch 11/60
910/910 [==============================] - 529s 581ms/step - loss: 0.1814 - acc: 0.9285 - val_loss: 0.0391 - val_acc: 0.9834
Epoch 12/60
910/910 [==============================] - 529s 581ms/step - loss: 0.1715 - acc: 0.9342 - val_loss: 0.6787 - val_acc: 0.7792
Epoch 13/60
910/910 [==============================] - 527s 579ms/step - loss: 0.1678 - acc: 0.9361 - val_loss: 0.0451 - val_acc: 0.9796
Epoch 14/60
910/910 [==============================] - 529s 581ms/step - loss: 0.1683 - acc: 0.9356 - val_loss: 0.7874 - val_acc: 0.7306
Epoch 15/60
910/910 [==============================] - 528s 580ms/step - loss: 0.1618 - acc: 0.9387 - val_loss: 0.0483 - val_acc: 0.9761
Epoch 16/60
910/910 [==============================] - 528s 581ms/step - loss: 0.1569 - acc: 0.9398 - val_loss: 0.9105 - val_acc: 0.7060
Epoch 17/60
910/910 [==============================] - 527s 579ms/step - loss: 0.1566 - acc: 0.9397 - val_loss: 0.0380 - val_acc: 0.9853
Epoch 18/60
910/910 [==============================] - 529s 581ms/step - loss: 0.1506 - acc: 0.9416 - val_loss: 0.7649 - val_acc: 0.7435
Epoch 19/60
910/910 [==============================] - 527s 580ms/step - loss: 0.1497 - acc: 0.9429 - val_loss: 0.0507 - val_acc: 0.9778
Epoch 20/60
910/910 [==============================] - 529s 581ms/step - loss: 0.1476 - acc: 0.9439 - val_loss: 0.7189 - val_acc: 0.7665
Epoch 21/60
910/910 [==============================] - 527s 579ms/step - loss: 0.1426 - acc: 0.9447 - val_loss: 0.0377 - val_acc: 0.9873
Epoch 22/60
910/910 [==============================] - 528s 580ms/step - loss: 0.1407 - acc: 0.9463 - val_loss: 0.7066 - val_acc: 0.7817
Epoch 23/60
910/910 [==============================] - 526s 578ms/step - loss: 0.1427 - acc: 0.9444 - val_loss: 0.0376 - val_acc: 0.9877
Epoch 24/60
910/910 [==============================] - 528s 580ms/step - loss: 0.1373 - acc: 0.9467 - val_loss: 0.6619 - val_acc: 0.8023
Epoch 25/60
910/910 [==============================] - 528s 580ms/step - loss: 0.1362 - acc: 0.9466 - val_loss: 0.0457 - val_acc: 0.9844
Epoch 26/60
910/910 [==============================] - 529s 582ms/step - loss: 0.1350 - acc: 0.9474 - val_loss: 0.8683 - val_acc: 0.7046
Epoch 27/60
910/910 [==============================] - 527s 579ms/step - loss: 0.1339 - acc: 0.9492 - val_loss: 0.0411 - val_acc: 0.9855
Epoch 28/60
910/910 [==============================] - 529s 581ms/step - loss: 0.1339 - acc: 0.9499 - val_loss: 0.9552 - val_acc: 0.6762
Epoch 29/60
910/910 [==============================] - 527s 579ms/step - loss: 0.1343 - acc: 0.9488 - val_loss: 0.0446 - val_acc: 0.9859
Epoch 30/60
910/910 [==============================] - 528s 580ms/step - loss: 0.1282 - acc: 0.9513 - val_loss: 0.8127 - val_acc: 0.7298
Epoch 31/60
910/910 [==============================] - 527s 579ms/step - loss: 0.1286 - acc: 0.9504 - val_loss: 0.0484 - val_acc: 0.9857
Epoch 32/60
910/910 [==============================] - 529s 581ms/step - loss: 0.1258 - acc: 0.9506 - val_loss: 0.5007 - val_acc: 0.8479
Epoch 33/60
910/910 [==============================] - 527s 579ms/step - loss: 0.1301 - acc: 0.9495 - val_loss: 0.0467 - val_acc: 0.9859
Epoch 34/60
910/910 [==============================] - 529s 581ms/step - loss: 0.1253 - acc: 0.9516 - val_loss: 0.6061 - val_acc: 0.8056
Epoch 35/60
910/910 [==============================] - 527s 579ms/step - loss: 0.1259 - acc: 0.9521 - val_loss: 0.0469 - val_acc: 0.9873
Epoch 36/60
910/910 [==============================] - 528s 580ms/step - loss: 0.1249 - acc: 0.9511 - val_loss: 0.8658 - val_acc: 0.7121
Epoch 37/60
910/910 [==============================] - 527s 579ms/step - loss: 0.1206 - acc: 0.9548 - val_loss: 0.0459 - val_acc: 0.9869
Epoch 38/60
910/910 [==============================] - 527s 580ms/step - loss: 0.1229 - acc: 0.9512 - val_loss: 0.4516 - val_acc: 0.8646
Epoch 39/60
910/910 [==============================] - 527s 579ms/step - loss: 0.1206 - acc: 0.9528 - val_loss: 0.0469 - val_acc: 0.9861
Epoch 40/60

The below graph is not for this problem but similar situation of what I'm asking about:

loss

acc

$\endgroup$
0
$\begingroup$

Have you confirmed the variables batch_size and val_batch_size in your code contain the same value? Also consider checking that nb_validation_samples is correct.

If the value you pass into the validation_steps argument is not equal to the total number of validation batches of data you have, you'll end up validation your model on different batches of data each time, which (might) create the pattern you see.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.