How can I train my deep learning model on another similar yet different dataset

Question

I am doing semantic segmentation (multi-class classification of image pixels) using convolutional neural networks (CNN) in Keras.

In particular, I am applying this to aerial images of crops (vegetation). In Keras, I successfully developed a workflow to segment/classify different crops for one specific dataset (let's call this dataset rural area #1).

Can I apply my Keras weights trained on rural area #1 for initializing the training of another dataset rural area #2? Such as:

model = load_model("weights_ruralarea1.hdf5")

Then I will proceed to model.fit in Keras.

The rural area #2 dataset has a little amount of training images for training the CNN.

The images in this 2nd dataset, although it has similar crop content on the images to the 1st dataset, it also has different image resolution, and is not exactly the same visually as the 1st dataset.

So would using my weights for rural area #1 be a form of transfer learning? or will it be a form of fine-tuning?

Answer 1

1) If you have little to no data in the second dataset, the best you can do is use the first one. If you have "some" data you backpropogate through the last layer, if you have "some more" data, you back propagate through the last 2 layers. Using a well trained model could be good enough in itself (For instance, ResNet could be used right out the box with no tuning whatsoever

2) For a different size image, I think you could just reshape the image to that in dataset1. See https://keras.io/applications/#usage-examples-for-image-classification-models, for an example where the image is reshaped to 224 x 224 to fit the ResNet50 needs

Answer 2

When you train a CNN on rural area #1 dataset, I am assuming you have a fully connected(FC) layer towards the end. When you transfer the weights to the new network the best way to do would be to assign all the weights except for that of dense layer. The FC layer of the previously trained network with n fully connected nodes would have a weight matrix of size[m X n]. This layer will expect the input to it to be of size m. However due to the change in image size you will end up with a different value for m when you feed the image from the new dataset(convolution fiter convolving on a different image size).

So you have a couple of options here:

1. As Sid mentioned, you can resize the new image into the same dimension as the other dataset provided you do not have any scale relevant features in the data.
2. Remove the fully connected layers of the trained model and keep the conv-pooling layers. Add a new dense layer, keeping the conv-pooling layer weights fixed, learn the weights for the FC layer on the new data.

If you do not have an FC layer, it should be invariant of the input image dimension.

b) To answer your second question, in a way you are doing transfer learning as you are transferring the model trained for a different classification problem. You would do fine tuning by training the last few layers of this loaded model.