I haven't seen this question addressed anywhere. Hopefully my notation isn't too nonstandard, I'm a newcomer to the area of artificial neural networks.

Suppose we are training a neural network to classify data into one of $K$ classes, $\hat{y}_i \in \{1,2,\ldots, K\}$, on the basis of data $x_i$. So, we train a vector-valued function $f(x_i)$ which outputs scores (output layer activations) $f(x_i)=(s_{i,1},s_{i,2},\ldots,s_{i,K})$. We assign $\hat{y}_i$ on the basis of these scores, presumably to the class with the highest score. Further, we are using the cross-entropy loss:

$$ L_i(x_i , y_i) = -\log\Big( \dfrac{\exp(s_{i,k})}{\sum_{j=1}^{K}\exp(s_{i,j})} \Big) $$

where $y_i = k$; $k$ is the true label of the response. The full loss function is the mean of $L_1, \ldots, L_n$ assuming, as I am for this post, no regularization.

My question is this. If $K>2$, do we need to train $K$ different sets of weights to predict $K$ different scores $s_k$? Or, can we train only $K-1$ sets and let the score for the last class be determined? Everyone makes this clear for $K=2$ where we only need to output one score, and the score for the other class is defined by interpreting the scores as probabilities: $$P(y_i = 2 \mid x_i) = s_2 \implies s_1 = 1-s_2$$. But it is not obvious what the best thing to do is with $K > 2$. Furthermore, without knowing how to define $s_{i,K}$, I cannot compute the cross-entropy loss.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.