Alternative ways of coding the same output in neural networks? When using a neural network for supervised learning, say recognition of hand written digits, there are several ways to use the output layer to code for the expected output. I was wondering if there are any crucial differences in performance or what the overall tendencies are.
Basically, as a specific example with number recognition, say we have the output set of integrals {0,1,2,3}. I could code this using 4 output units, for which each single activation corresponds to one of the numbers (for example: 1000 = 0, 0100 = 1, 0010 = 2, 0001 = 3).  
I could also code this in binary using just 2 output layers, and train the network for giving the following output : 00 = 0, 01 = 1 , 10 = 2, 11 = 3. 
Are there any drawbacks of using this kind of architecture, where different outputs activate more than one output unit simultaneously?
 A: For your record, this is called normalizing/standarizing (not coding). 
Using binary is not the worst way to do this in my opinion. It adds a lot more non-linearity to your model, that will take more backpropagation (or more neurons) to be figured out.
It's quite illogical for a neural network that 1 = 01 and 2 = 10, but 3 = 11. The step from 2>3 is linear, but the step from 1>2is very complicated as it requires the outputs to be 'switched'. Even just dividing the outputs (0=0, 1=0.33, 2=0.67, 3=1) is more linear. Adding binary encoding only makes the task more complicatd.
Also, outputs will never be given perfectly rounded by a neural network. What if you have the output [0.34, 0.23], will you decode this as 0 = 00 or as 2 = 10. Both are feasable.
Using one-hot encoding is the way to go. Not only is this easier for a network to learn, but it also tells you the 2nd correct answer:
E.g. your output is [0.4, 0.93, 0.75, 0.1], this tells you that the handwritten digit is most likely a 1, but second most likely a 2. Binary encoding does not tell you any of this information.
