Currently, I am using a neural network to classify data into one of three groups (a logistic activation function is used on all but the output nodes). I can train the neural network in two ways:

1) For each observation $X_{i}$ I can have one output variable $O_{i}$ that can take three values: $1,2,$ or $3$.


2) For each observation $X_{i}$ I can have three output variables $O_{i1},O_{i2}$, and $O_{i3}$ that would take on the values $0$ or $1$ based on what the class label for that observation is. So for example, if observation $i$ belongs to class 2, $O_{i2}$ would be $1$, whereas $O_{i1}$ and $O_{i3}$ would be $0$.

Are the above two approaches equivalent? Is one better than the other? Are there any theoretical reasons to believe one should produce better results than the other?


Usually when you design a learning process with neural nets, you have to be aware of any structure you induce. This induced structure might be learned by net, since neural nets are very capable of incorporating patterns. The easiest way to induce a desired or undesired structure (learning bias) in the learning process is to manipulate accordingly the inputs and outputs.

Without further knowledge on your specific problem I suspect that the first option will induce a ordering relation between the outputs. What it means is that the neural net might probably "consider" that the distance between $O_1$ and $O_3$, than the distance between $O_1$ and $O_2$. If there is such an ordering, than you can proceed with this way. Anyway, it might be possible to have to scale the outputs as well, so the final learned values to not be $1, 2, 3$, but smaller values centered at $0$.

However, usually there are no such ordering relations between classes, so the second option "breaks" this possible induced structure. Each option has the same chances to be learned as any other. Going further with this rationale, I believe you have to also consider using softmax node outputs, which have the nice feature that the obtained output values comes from a probability function (the values are in $[0, 1]$ and the sum of the outputs is $1 = \sum_{j=1}^{3}O_{ij}$). For softmax nodes you can see how they are working on wikipedia

  • $\begingroup$ That makes sense; approach one would work for ordinal data whereas approach two would be more suited for nominal data. Thanks for the insight! (Would upvote but not enough karma). $\endgroup$ – dlaser Feb 18 '14 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.