4
$\begingroup$

I am using below small 3D CNN to predict whether 32*32*32 image cube in a CT scan is malignant or not.

def classifier(input_shape, kernel_size, pool_size):
    model = Sequential()

    model.add(Convolution3D(16, kernel_size[0], kernel_size[1], kernel_size[2],
                            border_mode='valid',
                            input_shape=input_shape))
    model.add(Activation('relu'))
    model.add(MaxPooling3D(pool_size=pool_size))
    model.add(Convolution3D(32, kernel_size[0], kernel_size[1], kernel_size[2]))
    model.add(Activation('relu'))
    model.add(MaxPooling3D(pool_size=pool_size))
    model.add(Convolution3D(64, kernel_size[0], kernel_size[1], kernel_size[2]))
    model.add(Activation('relu'))
    model.add(MaxPooling3D(pool_size=pool_size))
    model.add(Dropout(0.25))

    model.add(Flatten())
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(128))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(2))
    model.add(Activation('softmax'))

    return model

Compilation is as below

input_shape = (1,32,32,32)
model = classifier(input_shape, (3, 3, 3), (2, 2, 2))
model.compile(loss='categorical_crossentropy',
      optimizer=Adam(lr=1.0e-7),
      metrics=['accuracy'])

But when training from 2nd epoch the validation accuracy reach to 1.0000. as in below graphs. I tried with several learning rates (which sudden drops indicates restarting the training with reduced learning rates). enter image description here

I am using 900 image cube, label pairs as the training data set and 283 cube,label pairs as the validation set. Each cube is shifted 48 ways to increase the data set. I did this manually because Keras ImageDataGenerator do not support for 3D data augmentation. Therefore total no of train data =43116(~900*48) and total no of test data =13584 (~283*48). Data was standardized before fed to the network. My question is, is this due to this architecture doesn't suit for my classification problem or is this due to fewer no of data samples. Or is this is a indication of over-fitting? Can you please help me to figure out what is going wrong here.

$\endgroup$
4
  • 4
    $\begingroup$ 1.00 accuracy on validation is usually too good to be true, but it is not over-fitting (assuming your validation is not included in the training data). I'd start by checking the following: (1) What is the class imbalance in the data? Could it be that 99.9999% of the data is one class? (2) What features are meaningful in the model? Could it be that you inserted the true label to the images accidently? (3) Is your validation data really outside the train data? $\endgroup$
    – tmrlvi
    Jan 15 '18 at 8:28
  • 1
    $\begingroup$ This really looks a bug in validation set $\endgroup$
    – Maxim
    Jan 15 '18 at 9:18
  • 2
    $\begingroup$ As others have said, this sounds to be too good to be true. Aren't there any data leaks? Are training and validation sets kept separated? Maybe there is more subtle leak, e.g. in recent report they described how super accurate classifier build on x-ray images learned to detect rulers and medical on photos containing tumors, that were absent on the "negative" images. Are all the properties of positive and negative images the same (size, quality, angles, colors, etc.)? $\endgroup$
    – Tim
    Dec 11 '18 at 9:42
  • 3
    $\begingroup$ For quick and dirty check: randomly shuffle the labels, then split the data to train/validation and train - what happens? $\endgroup$
    – Tim
    Dec 11 '18 at 9:43
0
$\begingroup$

Possible reasons include small number of samples in validation data, easy dataset(very unlikely to happen here) and sometimes unshuffled training data. You could fix the problem by increasing your training data samples or augmenting your training data in order to create enough samples for the model to learn/validate.

Data augmentation docs

$\endgroup$
0
$\begingroup$

It is very impossible that you have imbalanced data and all data in the validation set is one class.

Think about fraud detection, most transactions are not fraud, and if the classifier just simply say non-fraud, the model can get very high accuracy. In addition, if the validation set does not have fraud cases, then the accuracy can be 1.0.

I assume for medical data, most cases are not malignant, and you may have limited data points. (As you mentioned you only have about 1000 images.)

$\endgroup$
-1
$\begingroup$

This means that your dataset are very easy to learn, so the CNN was able to get 1.0 validation accuracy because you are saying you are sure that validation data is not in training data.

Note that this is only data you have generated yourself, so that their images are "too" clean to learn. If you try to test your model on real samples, I think the accuracy will not be good at all.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.