0
$\begingroup$

SO, I've been working on the MICCAI BRATS 2016 dataset, and was trying to implement a state-of-the-art but very simple model using keras, which he originally did using theano.

Here's my model's code in keras:

K.set_image_dim_ordering('th')
    model = Sequential()


    #first set of CONV => CONV => CONV => LReLU => MAXPOOL
    model.add(Convolution2D(64, kernel_size=(3, 3), padding="same", data_format='channels_first', input_shape = (d, h, w), kernel_initializer = 'glorot_normal', bias_initializer=constant(0.1) ))
    model.add(LeakyReLU(alpha=alp))
    model.add(Convolution2D(64, kernel_size=(3, 3), border_mode="same", data_format='channels_first', input_shape = (64, 33, 33), kernel_initializer = 'glorot_normal', bias_initializer=constant(0.1) ))
    model.add(LeakyReLU(alpha=alp))
    model.add(Convolution2D(64, kernel_size=(3, 3), border_mode="same", data_format='channels_first', input_shape = (64, 33, 33), kernel_initializer = 'glorot_normal', bias_initializer=constant(0.1) ))
    model.add(LeakyReLU(alpha=alp))
    model.add(MaxPooling2D(pool_size=(3, 3), strides=(2, 2)))

    #second set of CONV => CONV => CONV => LReLU => MAXPOOL
    model.add(Convolution2D(128, kernel_size=(3, 3), border_mode="same", data_format='channels_first', input_shape = (64, 16, 16), kernel_initializer = 'glorot_normal', bias_initializer=constant(0.1) ))
    model.add(LeakyReLU(alpha=alp))
    model.add(Convolution2D(128, kernel_size=(3, 3), border_mode="same", data_format='channels_first', input_shape = (128, 16, 16), kernel_initializer = 'glorot_normal', bias_initializer=constant(0.1) ))
    model.add(LeakyReLU(alpha=alp))
    model.add(Convolution2D(128, kernel_size=(3, 3), border_mode="same", data_format='channels_first', input_shape = (128, 16, 16), kernel_initializer = 'glorot_normal', bias_initializer=constant(0.1) ))
    model.add(LeakyReLU(alpha = alp))
    model.add(MaxPooling2D(pool_size=(3, 3), strides=(2, 2)))

    #Fully connected layers

    # FC => LReLU => FC => LReLU
    model.add(Flatten())
    model.add(Dense(256, kernel_initializer = 'glorot_normal', bias_initializer=constant(0.1)))
    model.add(LeakyReLU(alp))
    model.add(Dropout(dropout))
    model.add(Dense(256, kernel_initializer = 'glorot_normal', bias_initializer=constant(0.1)))
    model.add(LeakyReLU(alp))
    model.add(Dropout(dropout))

    # FC => SOFTMAX
    model.add(Dense(classes, kernel_initializer = 'glorot_normal', bias_initializer = constant(0.1)))
    model.add(Activation("softmax"))

After I compile and run it, with a data of 20,000 (18000 training:2000 validation), I get the following graphs:

accuracy graphs

As far I as checked, I think the model was correctly replicated. Can you comment on what is causing this overfitting?

$\endgroup$
1
$\begingroup$

It's totally normal for training loss to keep going down, even though validation loss levels off. Dropout and other types of regularization, including capacity regularization (reducing numbers of layers/neurons etc) can mitigate this effect somewhat, but you're always going to be able to overtrain/overfit somewhat on the training set relative to the test/validation set.

The only thing you can try to do is mitigate this effect, eg using dropout, but you're already using dropout, so seems good.

Your dataset is pretty small (10s of thousands of samples), so the possibility to overfit is pretty high.

If you really want to reduce this effect you can reduce the capacity of your network etc, but, really, it's not like your graphs show evidence that the overfitting/overtraining is negatively affecting your ability to generalize to your validation set, so seems not really a major concern, on its own.

$\endgroup$
  • $\begingroup$ But if you look at the paper, the results are far better. My predictions are noisy and nowhere like they should be. $\endgroup$ – V Shreyas Mar 25 '17 at 18:39
  • $\begingroup$ Also, when I increase the data for training, my prediction becomes really noisy $\endgroup$ – V Shreyas Mar 25 '17 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.