0
$\begingroup$

I have a binary classification problem. I get ROI mammogram images and then apply a decomposition algorithm and as output I get 5 images which summation of them results in the original image. Now, having these five components, I apply a VGG-like deep model for training a network, considering an input shape as (320,320,5) in which I feed five components as different input channels. Below is the code for my network:

def MyNet():
    from keras.optimizers import Adam
    regularization_rate = 0.01
    model = Sequential()
    model.add(Conv2D(input_shape=(320, 320, 5), filters=64, kernel_size=(3, 3), padding="same", activation="relu"))
    model.add(BatchNormalization())
    model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2, 2), strides=(2, 2)))

    # Block 2
    model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2, 2), strides=(2, 2)))

    # Block 3
    model.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2, 2), strides=(2, 2)))

    # Block 4
    model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2, 2), strides=(2, 2)))

    # Block 5
    model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
    model.add(BatchNormalization())
    model.add(MaxPooling2D((2, 2), strides=(2, 2)))

    # Flatten and fully connected layers
    model.add(Flatten())
    model.add(Dense(4096, activation='relu', kernel_regularizer=l2(regularization_rate)))
    model.add(BatchNormalization())
    model.add(Dense(4096, activation='relu', kernel_regularizer=l2(regularization_rate)))
    model.add(BatchNormalization())

    # Output layer for binary classification
    model.add(Dense(1, activation='sigmoid'))

    # Compile the model
    model.compile(optimizer='adam', loss='binary_crossentropy',
                  metrics=['accuracy', 'AUC', f1_m, recall_m, precision_m])
    return (model)

And below is the code for training and validation using 5-fold cross validation:

def kfold_indices(data, k):
    fold_size = len(data) // k
    indices = np.arange(len(data))
    folds = []
    for i in range(k):
        val_indices = indices[i * fold_size: (i+1) * fold_size]
        train_indices = np.concatenate([indices[:i * fold_size], indices[(i+1) * fold_size:]])
        folds.append((train_indices, val_indices))
    return folds


k = 5
fold_indices = kfold_indices(X_train, k)
accuracies = []
index_fold = 1

for train_indices, val_indices in fold_indices:
    print("===============================", index_fold, "===========================================")
    X_train_fold, y_train_fold = X_train[train_indices], y_train[train_indices]

    X_val, y_val = X_train[val_indices], y_train[val_indices]

    # fittiing the model  on X_train and y_train --> K-fold
    model = MyNet()
    # model.compile(loss='binary_crossentropy', optimizer='RMSprop', metrics=['accuracy', 'AUC'])
    train_res = model.fit(X_train_fold, y_train_fold, epochs=50, batch_size=32, validation_data=(X_val, y_val),
                          validation_batch_size=32, verbose=1, callbacks=[temp_logger])
    accuracies.append(train_res)
    index_fold = index_fold + 1

The input X_train is in shape (1303, 320, 320, 5). The output is all folds are quite similar as below for the last 10 epochs:

Epoch 40/50
33/33 [==============================] - 10s 297ms/step - loss: 1.5994 - accuracy: 0.9310 - auc: 0.9771 - f1_m: 0.9352 - recall_m: 0.9474 - precision_m: 0.9288 - val_loss: 2.3136 - val_accuracy: 0.5385 - val_auc: 0.5847 - val_f1_m: 0.4224 - val_recall_m: 0.3604 - val_precision_m: 0.5441
Epoch 41/50
33/33 [==============================] - 10s 296ms/step - loss: 1.7654 - accuracy: 0.9060 - auc: 0.9602 - f1_m: 0.9013 - recall_m: 0.9089 - precision_m: 0.9058 - val_loss: 3.3083 - val_accuracy: 0.5231 - val_auc: 0.5456 - val_f1_m: 0.5832 - val_recall_m: 0.7177 - val_precision_m: 0.5242
Epoch 42/50
33/33 [==============================] - 10s 296ms/step - loss: 1.9983 - accuracy: 0.8974 - auc: 0.9504 - f1_m: 0.9001 - recall_m: 0.9098 - precision_m: 0.8982 - val_loss: 3.3215 - val_accuracy: 0.5115 - val_auc: 0.6062 - val_f1_m: 0.3833 - val_recall_m: 0.3506 - val_precision_m: 0.6310
Epoch 43/50
33/33 [==============================] - 10s 297ms/step - loss: 1.5076 - accuracy: 0.9291 - auc: 0.9785 - f1_m: 0.9303 - recall_m: 0.9401 - precision_m: 0.9278 - val_loss: 3.3200 - val_accuracy: 0.5346 - val_auc: 0.6354 - val_f1_m: 0.4369 - val_recall_m: 0.3833 - val_precision_m: 0.6552
Epoch 44/50
33/33 [==============================] - 10s 300ms/step - loss: 1.3974 - accuracy: 0.9348 - auc: 0.9816 - f1_m: 0.9363 - recall_m: 0.9352 - precision_m: 0.9443 - val_loss: 2.7090 - val_accuracy: 0.5846 - val_auc: 0.6078 - val_f1_m: 0.6341 - val_recall_m: 0.7522 - val_precision_m: 0.5820
Epoch 45/50
33/33 [==============================] - 10s 298ms/step - loss: 1.3208 - accuracy: 0.9185 - auc: 0.9742 - f1_m: 0.9194 - recall_m: 0.9292 - precision_m: 0.9169 - val_loss: 1.8723 - val_accuracy: 0.6538 - val_auc: 0.6667 - val_f1_m: 0.6646 - val_recall_m: 0.7490 - val_precision_m: 0.6422
Epoch 46/50
33/33 [==============================] - 10s 296ms/step - loss: 1.1986 - accuracy: 0.9482 - auc: 0.9866 - f1_m: 0.9498 - recall_m: 0.9463 - precision_m: 0.9584 - val_loss: 2.1230 - val_accuracy: 0.6346 - val_auc: 0.7082 - val_f1_m: 0.6837 - val_recall_m: 0.8356 - val_precision_m: 0.6041
Epoch 47/50
33/33 [==============================] - 10s 297ms/step - loss: 1.4068 - accuracy: 0.9367 - auc: 0.9869 - f1_m: 0.9380 - recall_m: 0.9429 - precision_m: 0.9421 - val_loss: 4.4375 - val_accuracy: 0.5885 - val_auc: 0.6231 - val_f1_m: 0.5886 - val_recall_m: 0.6213 - val_precision_m: 0.6146
Epoch 48/50
33/33 [==============================] - 10s 297ms/step - loss: 1.6808 - accuracy: 0.9070 - auc: 0.9637 - f1_m: 0.9073 - recall_m: 0.9208 - precision_m: 0.9032 - val_loss: 4.6520 - val_accuracy: 0.4846 - val_auc: 0.5533 - val_f1_m: 0.1789 - val_recall_m: 0.1086 - val_precision_m: 0.5926
Epoch 49/50
33/33 [==============================] - 10s 297ms/step - loss: 1.4291 - accuracy: 0.9291 - auc: 0.9765 - f1_m: 0.9305 - recall_m: 0.9386 - precision_m: 0.9301 - val_loss: 2.6339 - val_accuracy: 0.5885 - val_auc: 0.6236 - val_f1_m: 0.6426 - val_recall_m: 0.7774 - val_precision_m: 0.5738
Epoch 50/50
33/33 [==============================] - 10s 297ms/step - loss: 1.7352 - accuracy: 0.9003 - auc: 0.9643 - f1_m: 0.8992 - recall_m: 0.8958 - precision_m: 0.9141 - val_loss: 2.2339 - val_accuracy: 0.5885 - val_auc: 0.6073 - val_f1_m: 0.6624 - val_recall_m: 0.8475 - val_precision_m: 0.5640

As it can be seen there is a huge difference between training and validation results. I know there can be overfitting problem. As I searched through literature I found and tested the below options:

  1. adding dropout layer:modifying the last layers as
    model.add(Flatten())
    model.add(Dense(4096, activation='relu', kernel_regularizer=l2(regularization_rate)))
    model.add(BatchNormalization())
    model.add(Dropout(0.5))  # Add dropout with rate 0.5
    model.add(Dense(4096, activation='relu', kernel_regularizer=l2(regularization_rate)))
    model.add(BatchNormalization())
    model.add(Dropout(0.5))  # Add dropout with rate 0.5
  1. adding regularization:
model.add(Flatten())
    model.add(Dense(4096, activation='relu', kernel_regularizer=l2(regularization_rate)))
    model.add(BatchNormalization())
    model.add(Dense(4096, activation='relu', kernel_regularizer=l2(regularization_rate)))
    model.add(BatchNormalization())
  1. changing batch size: changing training batch size from 32 to 16 and 64
  2. changing loss function: changing from binary cross entropy to
tensorflow.keras.losses.BinaryFocalCrossentropy(gamma=2.0, from_logits=True)
  1. using different activation functions: tried with tanh

Doing all these, the results did not change for validation and in some cases it destroyed the training results as well.

What is the main reason behind this? Is the network design appropriate? and how should I address my problem?

I would really appreciate any help.

*A note, I have also tried with ResNet50. However, the results for training only changed not the validation. I have below the last five epochs results:

Epoch 45/50
33/33 [==============================] - 6s 183ms/step - loss: 2.8129e-05 - accuracy: 1.0000 - auc: 1.0000 - f1_m: 1.0000 - precision_m: 1.0000 - recall_m: 1.0000 - val_loss: 2.0236 - val_accuracy: 0.6077 - val_auc: 0.6410 - val_f1_m: 0.5755 - val_precision_m: 0.5821 - val_recall_m: 0.5809
Epoch 46/50
33/33 [==============================] - 6s 184ms/step - loss: 3.0869e-05 - accuracy: 1.0000 - auc: 1.0000 - f1_m: 1.0000 - precision_m: 1.0000 - recall_m: 1.0000 - val_loss: 2.0310 - val_accuracy: 0.6077 - val_auc: 0.6413 - val_f1_m: 0.5755 - val_precision_m: 0.5821 - val_recall_m: 0.5809
Epoch 47/50
33/33 [==============================] - 6s 183ms/step - loss: 2.9120e-05 - accuracy: 1.0000 - auc: 1.0000 - f1_m: 1.0000 - precision_m: 1.0000 - recall_m: 1.0000 - val_loss: 2.0403 - val_accuracy: 0.6077 - val_auc: 0.6413 - val_f1_m: 0.5755 - val_precision_m: 0.5821 - val_recall_m: 0.5809
Epoch 48/50
33/33 [==============================] - 6s 184ms/step - loss: 2.8139e-05 - accuracy: 1.0000 - auc: 1.0000 - f1_m: 1.0000 - precision_m: 1.0000 - recall_m: 1.0000 - val_loss: 2.0449 - val_accuracy: 0.6038 - val_auc: 0.6425 - val_f1_m: 0.5737 - val_precision_m: 0.5797 - val_recall_m: 0.5809
Epoch 49/50
33/33 [==============================] - 6s 184ms/step - loss: 2.9613e-05 - accuracy: 1.0000 - auc: 1.0000 - f1_m: 1.0000 - precision_m: 1.0000 - recall_m: 1.0000 - val_loss: 2.0428 - val_accuracy: 0.6077 - val_auc: 0.6423 - val_f1_m: 0.5755 - val_precision_m: 0.5821 - val_recall_m: 0.5809
Epoch 50/50
33/33 [==============================] - 6s 183ms/step - loss: 3.0790e-05 - accuracy: 1.0000 - auc: 1.0000 - f1_m: 1.0000 - precision_m: 1.0000 - recall_m: 1.0000 - val_loss: 2.0455 - val_accuracy: 0.6077 - val_auc: 0.6395 - val_f1_m: 0.5755 - val_precision_m: 0.5821 - val_recall_m: 0.5809

I am really stuck with this and any help would be really appreciated.

Improve this question
$\endgroup$
4
  • $\begingroup$ Possibly an issue to do with the optimizer. Maybe it found a local minimum. $\endgroup$
    – Joe King
    Commented Dec 14, 2023 at 21:17
  • $\begingroup$ @JoeKing Is there anyway to prevent this? $\endgroup$
    – Nmgh
    Commented Dec 14, 2023 at 21:39
  • $\begingroup$ Does this help: Why is there no improvement when training Xgboost with pseudo-Huber loss? $\endgroup$
    – Joe King
    Commented Dec 15, 2023 at 9:34
  • $\begingroup$ @JoeKing Thanks for sharing but no this is not related. $\endgroup$
    – Nmgh
    Commented Dec 15, 2023 at 15:32

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.