1
$\begingroup$

When using a neural network for supervised learning, say recognition of hand written digits, there are several ways to use the output layer to code for the expected output. I was wondering if there are any crucial differences in performance or what the overall tendencies are.

Basically, as a specific example with number recognition, say we have the output set of integrals {0,1,2,3}. I could code this using 4 output units, for which each single activation corresponds to one of the numbers (for example: 1000 = 0, 0100 = 1, 0010 = 2, 0001 = 3).

I could also code this in binary using just 2 output layers, and train the network for giving the following output : 00 = 0, 01 = 1 , 10 = 2, 11 = 3.

Are there any drawbacks of using this kind of architecture, where different outputs activate more than one output unit simultaneously?

$\endgroup$

1 Answer 1

3
$\begingroup$

For your record, this is called normalizing/standarizing (not coding).

Using binary is not the worst way to do this in my opinion. It adds a lot more non-linearity to your model, that will take more backpropagation (or more neurons) to be figured out.

It's quite illogical for a neural network that 1 = 01 and 2 = 10, but 3 = 11. The step from 2>3 is linear, but the step from 1>2is very complicated as it requires the outputs to be 'switched'. Even just dividing the outputs (0=0, 1=0.33, 2=0.67, 3=1) is more linear. Adding binary encoding only makes the task more complicatd.

Also, outputs will never be given perfectly rounded by a neural network. What if you have the output [0.34, 0.23], will you decode this as 0 = 00 or as 2 = 10. Both are feasable.

Using one-hot encoding is the way to go. Not only is this easier for a network to learn, but it also tells you the 2nd correct answer:

E.g. your output is [0.4, 0.93, 0.75, 0.1], this tells you that the handwritten digit is most likely a 1, but second most likely a 2. Binary encoding does not tell you any of this information.

$\endgroup$
2
  • $\begingroup$ Thank you for the answer, I will accept it. Most of it is clear and enlightening, but I fail to gain an intuition for the concept of linearity in the output. Could you explain why you describe the step 2 -> 3 to be linear, and the step 1 -> 2 not being linear? $\endgroup$
    – hirschme
    May 23, 2017 at 13:21
  • $\begingroup$ 1 is bigger than 0. When going from step 2>3, you add 1. So it's linear with the number. However when going from 1>2, you substract the 2nd 1, and you add a new 1 on the 1st output. This is a non-linear operation, it is not linear with 1>2 $\endgroup$ May 23, 2017 at 16:09

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .