0
$\begingroup$

When using a neural network for supervised learning, say recognition of hand written digits, there are several ways to use the output layer to code for the expected output. I was wondering if there are any crucial differences in performance or what the overall tendencies are.

Basically, as a specific example with number recognition, say we have the output set of integrals {0,1,2,3}. I could code this using 4 output units, for which each single activation corresponds to one of the numbers (for example: 1000 = 0, 0100 = 1, 0010 = 2, 0001 = 3).

I could also code this in binary using just 2 output layers, and train the network for giving the following output : 00 = 0, 01 = 1 , 10 = 2, 11 = 3.

Are there any drawbacks of using this kind of architecture, where different outputs activate more than one output unit simultaneously?

$\endgroup$
3
$\begingroup$

For your record, this is called normalizing/standarizing (not coding).

Using binary is not the worst way to do this in my opinion. It adds a lot more non-linearity to your model, that will take more backpropagation (or more neurons) to be figured out.

It's quite illogical for a neural network that 1 = 01 and 2 = 10, but 3 = 11. The step from 2>3 is linear, but the step from 1>2is very complicated as it requires the outputs to be 'switched'. Even just dividing the outputs (0=0, 1=0.33, 2=0.67, 3=1) is more linear. Adding binary encoding only makes the task more complicatd.

Also, outputs will never be given perfectly rounded by a neural network. What if you have the output [0.34, 0.23], will you decode this as 0 = 00 or as 2 = 10. Both are feasable.

Using one-hot encoding is the way to go. Not only is this easier for a network to learn, but it also tells you the 2nd correct answer:

E.g. your output is [0.4, 0.93, 0.75, 0.1], this tells you that the handwritten digit is most likely a 1, but second most likely a 2. Binary encoding does not tell you any of this information.

$\endgroup$
2
  • $\begingroup$ Thank you for the answer, I will accept it. Most of it is clear and enlightening, but I fail to gain an intuition for the concept of linearity in the output. Could you explain why you describe the step 2 -> 3 to be linear, and the step 1 -> 2 not being linear? $\endgroup$ – hirschme May 23 '17 at 13:21
  • $\begingroup$ 1 is bigger than 0. When going from step 2>3, you add 1. So it's linear with the number. However when going from 1>2, you substract the 2nd 1, and you add a new 1 on the 1st output. This is a non-linear operation, it is not linear with 1>2 $\endgroup$ – Thomas Wagenaar May 23 '17 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.