Multi Output Neural Networks

Question

Up until know I only used neural networks to classify a single output, I set one output neuron for each class and check which neuron has the highest/lowest activation.

What I am trying to do is to detect a pattern and instead of outputting a single value (either class or activation value) I would like to output multiple values. eg,

[0,5 0,5 0,5] -> [0,5 0,5 0,5]
[1 1 1] -> [1 1 1]
[2 2 2] -> [-1 -1 -1]

So what I am wondering is can I use a network with 3 outputs and instead of checking activation, use all outputs as my output pattern?

Answer 1

The technique you seem be trying to use is what is sometimes called GSM (General Softmax) output layer, described by (Dieterrich & Bakiri, 1995). In this technique you represent each class with a fixed binary code and train your network to output the right code for each training example. This has been reported to increase performance by reducing overfitting in cases in which there are a large number of classes and their distributions are far from uniform (e.g. Mohamed et Al., 2009).

This approach isn't very popular though. It's more usual to just use a regular Softmax as the output layer, which is just the generalization of logistic regression for the multiclass setting. In most problems it's likely that the differences between approaches wouldn't be significant.