1
$\begingroup$

I was reading Bishop's Pattern Recognition book, specifically, I was reading his notation for expressing the error just before back propagation. The particular equation I am a little confused about is the following:

enter image description here

In that section he discussed how there is an error measure for each data point specifically. I think that is what the subscript means for $E_n$. In that same section he clearly says what $y_{nk}$ means:

$$ y_{nk} = y_{k}(\bf{x}_n , \bf{w} )$$

With that information I tried to deduce/infer what $t_{nk}$ meant. I believe he usually uses $t$ to mean the target value, so I think thats the supervised label we are trying to learn. However, what confuses me is why it requires two subscripts, both an $n$ and $k$. I would assume that $n$ refers to the data point we want to learn from, but $k$ is a bit unclear to me. Also, something that bugs me about this notation that seems unclear is that every output unit $y_k$ have the same $w$, but that doesn't seem correct to me at all for a neural network. Shouldn't each output be a combination of the inputs? As in:

$$ y_{nk} = y_{k}(\bf{x}_n , \bf{w}_k )$$

where $\bf{w}_k$ is for each output?

He seems to fix this in equation 5.45:

enter image description here

so he did mean to give a subscript to each weight, right? As in $ y_{nk} = y_{k}(\bf{x}_n , \bf{w} )$?

Does it mean that the data set that we are trying to learn is $x_n \in \mathbb{R}^{d_x}$ $t_n, \in \mathbb{R}^{d_t}$ and the output of the network is a vector rather than a single number? I am not sure what I am confused about but I am guessing thats it.


Also, as a reference and add context I will add a bit more form the exact section that I am reading:

enter image description here

$\endgroup$
2
$\begingroup$

The final prediction of your neural net is going to be $y_{nk}=y_k(\mathbf{x_n},\mathbf{w})$, which is the output vector.

Now, each neuron at a hidden layer takes the input and multiplies it with the weights and adds the bias. Its actual activation $y_{nk}=y_k(\mathbf{x_n},\mathbf{w})$ could be rewritten as:

$$y_{nk}=\sigma(\sum_i w_{ki} x_{i} + b_{k})=\sigma(\mathbf{w_{k}} \mathbf{x_n} + b_{k}),$$

where $\mathbf{x_n}$ is either your input $n$ or the vector output of the previous layer, $\mathbf{w_{k}}$ are the weights of neuron $k$, and $\sigma$ is the activation function tanh, sigmoid etc. So, $\mathbf{w_{k}}$ and $\mathbf{x_n}$ are vectors. Vectorized forms are preferred for ease of readability.

Finally, $t_{nk}$ is the target (true) label of the $k$ class of the $n$th sample of your dataset that you are comparing your predictions in order to train the model. Therefore,

$$\frac{1}{2} \sum_k (y_{nk}-t_{nk})^2,$$

is the Mean Squared Error (MSE) of your predictions $y_{nk}$ compared to the target labels $t_{nk}$, and it is the most common loss criterion.

$\endgroup$
  • $\begingroup$ is the output a vector or not? What does $y_{nk} = y_k(x_n, w)$ mean? Does w implicitly is a data structure/vector whatever, with all the weights in the hidden units. Your answer is nearly just re-pasting what my question has in information. $\endgroup$ – Pinocchio Jun 14 '15 at 19:17
  • 1
    $\begingroup$ edited. $y_{nk}$ is your output scalar, and $\mathbf{y_{n}}$ is its vector form . Now, for ease of understadning $y_{nk}=y_k(\mathbf{x_n},\mathbf{w})$ could be rewritten as: $y_{nk}=\sigma(\sum_i w_{ki} x_{i} + b_{k})=\sigma(\mathbf{w_{k}} \mathbf{x_n} + b_{k})$, where $\mathbf{w_{k}}$ is the weight vector of neuron $k$ and $\mathbf{w_{k}} \mathbf{x_n}$ is a dot product so its a scalar. $\endgroup$ – Yannis Assael Jun 14 '15 at 19:54
0
$\begingroup$

I think you are confused with output representation. Correponding target to a sample is a vector rather than a scalar. The vector possess single 1 which correspond to true class and $ K-1$ '0's where $ K$ is number of classes. If n-th sample's k-th index, $ t_{nk} $, is 1 that means n-th sample belongs to class k. This type of representation is called one-hot. Here is a $ {\bf t}$ with N = 3, K = 5.

\begin{bmatrix} 1 & 0 & 0 & 0 & 0 \\[0.3em] 0 & 1 & 0 & 0 & 0 \\[0.3em] 0 & 0 & 0 & 1 & 0 \\[0.3em] \end{bmatrix}

sample 1,2,3 are belongs to class 1,2,4 respectively.

For your second question, matrix $ {\bf w}$ is used as an input to function $ y_k{\bf(x_n,w)}$, however only its k-th row is used to evaluate k-th output, $ y_{k}$, as you can see from eq. 5.45. You can also use $ {\bf w_k}$ as an input as long as you correct Eq.45 with respect to the new representation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.