What does the notation $t_{nk}$ mean for neural networks in Bishop's Pattern Recognition book?

Question

I was reading Bishop's Pattern Recognition book, specifically, I was reading his notation for expressing the error just before back propagation. The particular equation I am a little confused about is the following:

In that section he discussed how there is an error measure for each data point specifically. I think that is what the subscript means for $E_n$. In that same section he clearly says what $y_{nk}$ means:

$$ y_{nk} = y_{k}(\bf{x}_n , \bf{w} )$$

With that information I tried to deduce/infer what $t_{nk}$ meant. I believe he usually uses $t$ to mean the target value, so I think thats the supervised label we are trying to learn. However, what confuses me is why it requires two subscripts, both an $n$ and $k$. I would assume that $n$ refers to the data point we want to learn from, but $k$ is a bit unclear to me. Also, something that bugs me about this notation that seems unclear is that every output unit $y_k$ have the same $w$, but that doesn't seem correct to me at all for a neural network. Shouldn't each output be a combination of the inputs? As in:

$$ y_{nk} = y_{k}(\bf{x}_n , \bf{w}_k )$$

where $\bf{w}_k$ is for each output?

He seems to fix this in equation 5.45:

so he did mean to give a subscript to each weight, right? As in $ y_{nk} = y_{k}(\bf{x}_n , \bf{w} )$?

Does it mean that the data set that we are trying to learn is $x_n \in \mathbb{R}^{d_x}$ $t_n, \in \mathbb{R}^{d_t}$ and the output of the network is a vector rather than a single number? I am not sure what I am confused about but I am guessing thats it.

---

Also, as a reference and add context I will add a bit more form the exact section that I am reading:

Answer 1

The final prediction of your neural net is going to be $y_{nk}=y_k(\mathbf{x_n},\mathbf{w})$, which is the output vector.

Now, each neuron at a hidden layer takes the input and multiplies it with the weights and adds the bias. Its actual activation $y_{nk}=y_k(\mathbf{x_n},\mathbf{w})$ could be rewritten as:

$$y_{nk}=\sigma(\sum_i w_{ki} x_{i} + b_{k})=\sigma(\mathbf{w_{k}} \mathbf{x_n} + b_{k}),$$

where $\mathbf{x_n}$ is either your input $n$ or the vector output of the previous layer, $\mathbf{w_{k}}$ are the weights of neuron $k$, and $\sigma$ is the activation function tanh, sigmoid etc. So, $\mathbf{w_{k}}$ and $\mathbf{x_n}$ are vectors. Vectorized forms are preferred for ease of readability.

Finally, $t_{nk}$ is the target (true) label of the $k$ class of the $n$th sample of your dataset that you are comparing your predictions in order to train the model. Therefore,

$$\frac{1}{2} \sum_k (y_{nk}-t_{nk})^2,$$

is the Mean Squared Error (MSE) of your predictions $y_{nk}$ compared to the target labels $t_{nk}$, and it is the most common loss criterion.

Answer 2

I think you are confused with output representation. Correponding target to a sample is a vector rather than a scalar. The vector possess single 1 which correspond to true class and $ K-1$ '0's where $ K$ is number of classes. If n-th sample's k-th index, $ t_{nk} $, is 1 that means n-th sample belongs to class k. This type of representation is called one-hot. Here is a $ {\bf t}$ with N = 3, K = 5.

\begin{bmatrix} 1 & 0 & 0 & 0 & 0 \\[0.3em] 0 & 1 & 0 & 0 & 0 \\[0.3em] 0 & 0 & 0 & 1 & 0 \\[0.3em] \end{bmatrix}

sample 1,2,3 are belongs to class 1,2,4 respectively.

For your second question, matrix $ {\bf w}$ is used as an input to function $ y_k{\bf(x_n,w)}$, however only its k-th row is used to evaluate k-th output, $ y_{k}$, as you can see from eq. 5.45. You can also use $ {\bf w_k}$ as an input as long as you correct Eq.45 with respect to the new representation.