0
$\begingroup$

I am using below small 3D CNN to predict whether 32*32*32 image cube in a CT scan is malignant or not.

def classifier(input_shape, kernel_size, pool_size):
    model = Sequential()

    model.add(Convolution3D(16, kernel_size[0], kernel_size[1], kernel_size[2],
                            border_mode='valid',
                            input_shape=input_shape))
    model.add(Activation('relu'))
    model.add(MaxPooling3D(pool_size=pool_size))
    model.add(Convolution3D(32, kernel_size[0], kernel_size[1], kernel_size[2]))
    model.add(Activation('relu'))
    model.add(MaxPooling3D(pool_size=pool_size))
    model.add(Convolution3D(64, kernel_size[0], kernel_size[1], kernel_size[2]))
    model.add(Activation('relu'))
    model.add(MaxPooling3D(pool_size=pool_size))
    model.add(Dropout(0.25))

    model.add(Flatten())
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(128))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(2))
    model.add(Activation('softmax'))

    return model

Compilation is as below

input_shape = (1,32,32,32)
model = classifier(input_shape, (3, 3, 3), (2, 2, 2))
model.compile(loss='categorical_crossentropy',
      optimizer=Adam(lr=1.0e-7),
      metrics=['accuracy'])

But when training from 2nd epoch the validation accuracy reach to 1.0000. as in below graphs. I tried with several learning rates (which sudden drops indicates restarting the training with reduced learning rates). enter image description here

I am using 900 image cube, label pairs as the training data set and 283 cube,label pairs as the validation set. Each cube is shifted 48 ways to increase the data set. I did this manually because Keras ImageDataGenerator do not support for 3D data augmentation. Therefore total no of train data =43116(~900*48) and total no of test data =13584 (~283*48). Data was standardized before fed to the network. My question is, is this due to this architecture doesn't suit for my classification problem or is this due to fewer no of data samples. Or is this is a indication of over-fitting? Can you please help me to figure out what is going wrong here.

$\endgroup$
  • $\begingroup$ 1.00 accuracy on validation is usually too good to be true, but it is not over-fitting (assuming your validation is not included in the training data). I'd start by checking the following: (1) What is the class imbalance in the data? Could it be that 99.9999% of the data is one class? (2) What features are meaningful in the model? Could it be that you inserted the true label to the images accidently? (3) Is your validation data really outside the train data? $\endgroup$ – tmrlvi Jan 15 '18 at 8:28
  • 1
    $\begingroup$ This really looks a bug in validation set $\endgroup$ – Maxim Jan 15 '18 at 9:18
  • 1
    $\begingroup$ As others have said, this sounds to be too good to be true. Aren't there any data leaks? Are training and validation sets kept separated? Maybe there is more subtle leak, e.g. in recent report they described how super accurate classifier build on x-ray images learned to detect rulers and medical on photos containing tumors, that were absent on the "negative" images. Are all the properties of positive and negative images the same (size, quality, angles, colors, etc.)? $\endgroup$ – Tim Dec 11 '18 at 9:42
  • 1
    $\begingroup$ For quick and dirty check: randomly shuffle the labels, then split the data to train/validation and train - what happens? $\endgroup$ – Tim Dec 11 '18 at 9:43
0
$\begingroup$

This means that your dataset are very easy to learn, so the CNN was able to get 1.0 validation accuracy because you are saying you are sure that validation data is not in training data.

Note that this is only data you have generated yourself, so that their images are "too" clean to learn. If you try to test your model on real samples, I think the accuracy will not be good at all.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.