After training a classifier on N classes, how do you add an N+1th class?

Question

Say we have data for N classes and train a classifier. Then we have a new set of data for a N+1th class. How do I train a classifier that now predicts all N+1 classes?

Setting is in object detection, though I don't think it should be different between image classification and object detection. Also, I am finetuning from a pre-trained model (e.g. VGG16 trained on image-net).

Option 1: Compile all the data for the N+1 classes and finetune again from the original VGG16 trained on image-net.

Option 2: Compile all the data for the N+1 classes and finetune from the model trained on N classes.

Option 3: Is there a way to finetune only using the data from the N+1th class? I assume no, since the last layer's weights (right before class prediction) need to be re-initialized.

Options 1 and 2 aren't ideal b/c it isn't scalable as we increase the number of classes.

Answer 1

Another approach is to train autoencoders for your first N classes, and then train another network $f$ which maps trained autoencoder weights to vectors $W_i$, where $W_i$ is the $i$th row in the weight matrix in the FC layer before the softmax (this vector would correspond to the $i$th class being picked).

Now in order to add a new class, you can simply train an autoencoder on just that class, and use $f$ to predict the vector $W_{N+1}$. Now you can concatenate it onto your weight matrix, and perhaps fine-tune a bit.

This doesn't require retraining on all the data (although finetuning may improve performance), and the cost is constant with respect to $N$.

Also see the extremely relevant paper "Learning to Model the Tail"

Answer 2

Option 2.

Model generates (presumably) some kind of Softmax output in the last layer. You will have to train the network to distinguish class N + 1 from all other classes. So to the best of my knowledge, there's no magic to just incrementally train on new class.

I'd treat output of VGG as a unsupervised pre-processing of input. That need not be tuned again for new data classes. In other words, this is the goal of pre-trained models: unsupervised learning usable across applications.