Multi class classification using neural networks - where classes are added and removed frequently

Question

I have a classification problem where neural networks appear to be a promising choice (I don't get into details, because my question is about the general approach)

When using "classical" machine learning algorithms, a common solution for multi class classification is to train a classifier for each class (one vs. all) - this has the benefit of easily adding and removing classes.

Theoretically, this can also be done with neural networks, but then we loose the benefit of "reusing" the existing neural network that was trained for the other classes

are there any "best practices" for this type of problem?

My basic ideas are (they are basic ideas - not "production ready"):

1. use one vs all training, and pick an existing network as template
2. use a network with one output neuron for each class, and adding an output for each new class (which requires changing the network structure)
3. use an auto encoder that trains on all data, and build a "classic" classifier based on the output of the auto encoder hidden layer

Answer 1

I would use a single network with multiple outputs (instead of multiple single-output networks, see this question for the differences), with the following approach:

The whole network will be train with some hidden layers and final softmax layer as the output. Assuming that most of the learned features will be useful for multiple classes (which is the reason why we prefer a single network), when adding a new class, most of the network might remain fixed. The last hidden layer would be regarded as inputs to the last layer of the original network. So only a single layer should be trained when adding a new class.

Of course, it will be useful to allow training of the whole network for fine tuning, but as the number of classes is increased the importance of such fine tuning is decreased.