# Alternative ways of coding the same output in neural networks?

When using a neural network for supervised learning, say recognition of hand written digits, there are several ways to use the output layer to code for the expected output. I was wondering if there are any crucial differences in performance or what the overall tendencies are.

Basically, as a specific example with number recognition, say we have the output set of integrals {0,1,2,3}. I could code this using 4 output units, for which each single activation corresponds to one of the numbers (for example: 1000 = 0, 0100 = 1, 0010 = 2, 0001 = 3).

I could also code this in binary using just 2 output layers, and train the network for giving the following output : 00 = 0, 01 = 1 , 10 = 2, 11 = 3.

Are there any drawbacks of using this kind of architecture, where different outputs activate more than one output unit simultaneously?

For your record, this is called normalizing/standarizing (not coding).

Using binary is not the worst way to do this in my opinion. It adds a lot more non-linearity to your model, that will take more backpropagation (or more neurons) to be figured out.

It's quite illogical for a neural network that 1 = 01 and 2 = 10, but 3 = 11. The step from 2>3 is linear, but the step from 1>2is very complicated as it requires the outputs to be 'switched'. Even just dividing the outputs (0=0, 1=0.33, 2=0.67, 3=1) is more linear. Adding binary encoding only makes the task more complicatd.

Also, outputs will never be given perfectly rounded by a neural network. What if you have the output [0.34, 0.23], will you decode this as 0 = 00 or as 2 = 10. Both are feasable.

Using one-hot encoding is the way to go. Not only is this easier for a network to learn, but it also tells you the 2nd correct answer:

E.g. your output is [0.4, 0.93, 0.75, 0.1], this tells you that the handwritten digit is most likely a 1, but second most likely a 2. Binary encoding does not tell you any of this information.

• Thank you for the answer, I will accept it. Most of it is clear and enlightening, but I fail to gain an intuition for the concept of linearity in the output. Could you explain why you describe the step 2 -> 3 to be linear, and the step 1 -> 2 not being linear? – hirschme May 23 '17 at 13:21
• 1 is bigger than 0. When going from step 2>3, you add 1. So it's linear with the number. However when going from 1>2, you substract the 2nd 1, and you add a new 1 on the 1st output. This is a non-linear operation, it is not linear with 1>2 – Thomas Wagenaar May 23 '17 at 16:09