Convolutionalizing fully connected layers to form an FCN in Keras

Question

I trained a simple classifier to detect whether or not an image contains a lane line.

model = Sequential()

model.add(Reshape((image_size[1], image_size[0], 3), input_shape=input_shape))

model.add(Convolution2D(32, 3, 3,
                        border_mode='valid'))
model.add(Activation('relu'))
model.add(Convolution2D(32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=pool_size))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(2))
model.add(Activation('softmax'))

This classifier converged at an accuracy of 98%. Not bad!

After reading this paper, and some related topics on here, I replaced the fully connected (Dense) layers with convolutional ones.

model = Sequential()

#model.add(Reshape((image_size[1], image_size[0], 3), input_shape=input_shape))

model.add(Convolution2D(32, 3, 3, border_mode='same', input_shape=input_shape,
            name='conv1-1', activation='relu'))
model.add(Convolution2D(32, 3, 3, border_mode='same',
            name='conv1-2', activation='relu'))
model.add(MaxPooling2D(pool_size=pool_size))

'''
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(2))
model.add(Activation('softmax'))
'''

model.add(Dropout(0.25))
model.add(Convolution2D(256, 90, 160, activation='relu', name='conv7'))
model.add(Dropout(0.5))
model.add(Convolution2D(2, 1, 1))
model.add(Flatten())
model.add(Activation('softmax'))

This classifier converged at an accuracy of 49%. I would be better off flipping a coin. Based on what I've read, the two should be equivalent - a convolution over the entire input is the same thing as a fully connected layer.

I made three notable changes.

1. As stated, convolutionalizing the fully connected layers. I think I understand this but it's entirely possible I didn't implement it properly.
2. Removing the Reshape() layer at the beginning. I'm not sure why it was there to begin with - ImageDataGenerator already returns images in the desired format (height, width, channels).
3. Changing the neurons in the first fully connected layer / convolution over the entire input from 128 to 256.

Can anyone identify what I did wrong?

Answer 1

I did some experimenting with Keras' MNIST tutorial.

If I edit the model to be fully convolutional, then train it, I encounter the same problem.

If I instead train the model as written, save the weights, and then import them to a convolutionalized model (reshaping where appropriate), it tests as perfectly equivalent. However, training it further causes accuracy to drop drastically.

So changing the network to be fully convolutional changes the gradient in some way, such that the network no longer converges at an optimum. This page claims that there is some way to train a network as fully convolutional from the start, but does not say how. Possibly it involves the use of a different loss function.

For those interested, my code for convolutionalizing the MNIST tutorial and reimporting the weights is below.

from __future__ import print_function
import keras
from keras.utils import plot_model
from keras.datasets import mnist
from keras.models import Sequential, load_model
from keras.layers import Dense, Dropout, Flatten, Activation
from keras.layers import Conv1D, Conv2D, MaxPooling2D
from keras import backend as K
import numpy as np

*same as tutorial*

weights = load_model('CNN.h5').get_weights()

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape,
                 padding='same'))
model.add(Conv2D(64, (3, 3), activation='relu',
                 padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
#model.add(Flatten())
#model.add(Dense(128, activation='relu'))
model.add(Conv2D(128, (14,14), activation='relu', padding='valid'))
model.add(Dropout(0.5))
#model.add(Dense(num_classes, activation='softmax'))
model.add(Conv2D(num_classes, (1,1), activation='softmax'))
model.add(Flatten())
plot_model(model, 'model.png', show_shapes=True)
model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

model.layers[0].set_weights([weights[0], weights[1]])
model.layers[1].set_weights([weights[2], weights[3]])
model.layers[4].set_weights([weights[4].reshape([14,14,64,128]), weights[5]])
model.layers[6].set_weights([weights[6].reshape([1,1,128,num_classes]), weights[7]])

model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs,
          verbose=1, validation_data=(x_test, y_test))
score = model.evaluate(x_test, y_test, verbose=0)
#model.save('CNN.h5')

print('Test loss:', score[0])
print('Test accuracy:', score[1])

Answer 2

I reworked on the Keras MNIST example and changed the fully connected layer at the output with a 1x1 convolution layer. I got the same accuracy as the model with fully connected layers at the output.

# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

visible = Input(shape=(28, 28, 1))
layer = Conv2D(32, (3,3), activation='relu')(visible)
layer = Conv2D(32, (3,3), activation='relu')(layer)
layer = MaxPooling2D(pool_size=(2, 2))(layer)
layer = Dropout(0.25)(layer)
layer = Conv2D(64, (3,3), activation='relu')(layer)
layer = Conv2D(64, (3,3), activation='relu')(layer)
layer = MaxPooling2D(pool_size=(2, 2))(layer)
layer = Dropout(0.25)(layer)
layer = Conv2D(128, (3,3), activation='relu')(layer)


layer = Conv2D(128, (1,1), activation='relu')(layer)
out_dense = Conv2D(num_classes, (1,1), activation='softmax')(layer)
out_dense = GlobalAveragePooling2D()(out_dense)

model = Model(inputs=visible, outputs=out_dense)


model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer='adam',
              metrics=['accuracy'])