Say we were to have a multi-classification problem with 10 classes. Our feature extractor is ready and good to go.

We train the model and all is good, but how does the model know what the true target value for each output node is in order to minimize its error?

In other words, if we have 10 output nodes lined up on top of each other (visually), then which one of these 10 nodes would correspond to class 1? (and the same questions is posed for the rest of the classes ideally)

Is this association done by the NN automatically as it's training or is there something influencing the order in which the classes are being associated with the output nodes?


It really depends on how you set up your one-hot encoding.

For each set of inputs to the network $\textbf{x}_i$, you have a corresponding classification $\textbf{y}_i$ that's know in advance.

Essentially $\textbf{y}_i$ is a vector of 1's and 0's. If you had three classes (A, B and C), and you're interested in class A, you could represent this as $\textbf{y}_i = (1, 0, 0)$. Class B could be $\textbf{y}_i = (0, 1, 0)$, which then means class C has to be $\textbf{y}_i = (0, 0, 1)$. Another way to interpret this is that class A points to the first node in the output layer, class B points to the second node, and the class C points to third and final node.

However, there's nothing stopping you from saying class A is $\textbf{y}_i = (0, 1, 0)$, class B is $\textbf{y}_i = (0, 0, 1)$ and class C is $\textbf{y}_i = (1, 0, 0)$.

  • $\begingroup$ So are you saying that there is an inherent association between how you one hot encode your classes and what is class is associated with which output node? (i.e. is (1,0,0) always the first node?) Is this documented somewhere? $\endgroup$ – Kamal Raydan Feb 9 at 12:21
  • $\begingroup$ @KamalRaydan This is “documented “ by whatever loss function you’re using. A simple inspection of multinational cross entropy loss will make it clear why this is the case, for instance. $\endgroup$ – Sycorax Feb 9 at 13:54
  • 1
    $\begingroup$ @KamalRaydan What Sycorax said. You could think about it like this - if you represent class A as $(1, 0, 0)$, then the network will 'light up' the first node in the output whenever it thinks the input maps to class A. However, if you represented class A as $(0, 0, 1)$, then the network would light up the last node whenever it thinks the input maps to class A. $\endgroup$ – ralph Feb 11 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.