Add New Object Class in Deep Learning Network

Question

Assuming that I have a trained deep learning network that can detect 10 classes of objects (road, sky, tree, etc.) in images. It takes in RGB images and outputs a probability map of size (img_col, img_row, n_class), and the final segmentation will be a argmax operation over the last dimension.

Now I want to add a new class to the network, for example, pedestrians, so that after training, the network will be able to detect pedestrians in images.

But I no longer have the old training data. Instead, I've got a new dataset which also contains pedestrians, but only the pedestrians are labeled. Training with the new dataset (using the old weights as initialization) will be the most straightforward way, but I would like to hear some other approaches.

Could anyone share some thoughts on how to realize this?

Answer 1

You should, at least, re-train the classification layer. When you add another output, during learning the new class, the other class activation must shrink either explicitly (sigmoid) or implicitly (softmax). However, it may be better to learn, at least, last feature layer as there would be some useful features to recognize pedestrians.

Another approach can be feeding the new class to the network and collect confidence from the output. Low confidence can be indication of another class that is not belong to any of the classes learnt before. For sure this method can also give low confidence to another class other than pedestrian or any class the network learnt on. Also, NN is a non-local generalization method. It is prone to classify a totally garbage image with high confidence (See adversarial examples if you are curious).

Answer 2

A few years after the question was asked, there are several attempt to solve this problem.

My best guess would be to:

• add a class in the last layer

• train the class corresponding to pedestrian with the new data

• try not to move the way the network was predicting the other class by using distillation. This may need some other data than the pedestrian one, where other class appear (cars, see, microscope,... ). The good thing is that if you are able to sample these (maybe through internet), you don't have to label them to make the technique work. If the new data with pedestrian labels come from the same distribution as the one used to train the network initially, you don't even have to think about that.

An example of this technique can be found in this paper: https://arxiv.org/abs/1708.06977

You may find other relevant papers by searching the following themes: "Continuous learning", "Lifelong Learning", "Catastrophic Forgetting".

[EDITS]

I recently read and loved these related articles: learning without forgetting, iCaRL, and End-to-End Incremental Learning

Answer 3

In order to add a class you will almost certainly need a differently structured network (ie. +1 output). You may also require more hidden nodes or inputs depending on your problem. Of course, as you mentioned you could simply re-train the parameters based on all the new data, however, you will lose all the benefit of the original dataset.

On possibility would be to initialize your new network (or at least the same number of input/hidden/output parameters) with the weights you have from the original dataset and then train on the new data. This will almost certainly speed up the process and to some degree retains the original information from the first dataset assuming it generalized well enough.

Answer 4

I don't know if it would work, but one approach would be to add a new neuron in the output softmax layer and train with gradient descent but updating only the weights going to that new class.

It is probably a very sub-optimal method but it could be worth trying.