# Backpropagation algorithm and error in hidden layer

I got a slight confusion on the backpropagation algorithm used in multilayer perceptron (MLP).

The error is adjusted by the cost function. In backpropagation, we are trying to adjust the weight of the hidden layers. The output error I can understand, that is, e = d - y [Without the subscripts].

The questions are:

1. How does one get the error of hidden layer? How does one calculate it?
2. If I backpropagate it, should I use it as a cost function of an adaptive filter or should I use a pointer (in C/C++) programming sense, to update the weight?
• NN is rather an obsolete technology, so I'm afraid you won't get an answer because no one here is using them...
– user88
Dec 11, 2010 at 12:55
• @mbq: I do not doubt your words, but how do you come to the conclusion that NN are "obsolete technology"? Dec 12, 2010 at 11:15
• @steffen By observation; I mean it is obvious that no one significant from NN community will come out and say "Hey guys, let's drop our life work and play with something better!", but we have tools that achieve same or better accuracy without all this ambivalence and never-ending training. And people do drop NN in favor of them.
– user88
Dec 13, 2010 at 9:18
• This had some truth when you said it, @mbq , but not anymore. Mar 9, 2013 at 2:18
• @jerad Pretty easy -- I simply haven't yet seen any fair comparison with other methods (Kaggle is not a fair comparison because of the lack of confidence intervals for accuracies -- especially when the results of all high-scoring teams are so close as in the Merck contest), neither any analysis of robustness of parameter optimisation -- which is way worse.
– user88
Mar 13, 2013 at 15:37

I figured I'd answer a self-contained post here for anyone that's interested. This will be using the notation described here.

# Introduction

The idea behind backpropagation is to have a set of "training examples" that we use to train our network. Each of these has a known answer, so we can plug them into the neural network and find how much it was wrong.

For example, with handwriting recognition, you would have lots of handwritten characters alongside what they actually were. Then the neural network can be trained via backpropagation to "learn" how to recognize each symbol, so then when it's later presented with an unknown handwritten character it can identify what it is correctly.

Specifically, we input some training sample into the neural network, see how good it did, then "trickle backwards" to find how much we can change each node's weights and bias to get a better result, and then adjust them accordingly. As we continue to do this, the network "learns".

There are also other steps that may be included in the training process (for example, dropout), but I will focus mostly on backpropagation since that's what this question was about.

# Partial derivatives

A partial derivative $$\frac{\partial f}{\partial x}$$ is a derivative of $$f$$ with respect to some variable $$x$$.

For example, if $$f(x, y)=x^2 + y^2$$, $$\frac{\partial f}{\partial x}=2x$$, because $$y^2$$ is simply a constant with respect to $$x$$. Likewise, $$\frac{\partial f}{\partial y}= 2y$$, because $$x^2$$ is simply a constant with respect to $$y$$.

A gradient of a function, designated $$\nabla f$$, is a function containing the partial derivative for every variable in f. Specifically:

$$\nabla f(v_1, v_2, ..., v_n) = \frac{\partial f}{\partial v_1 }\mathbf{e}_1 + \cdots + \frac{\partial f}{\partial v_n }\mathbf{e}_n$$,

where $$e_i$$ is a unit vector pointing in the direction of variable $$v_1$$.

Now, once we have computed the $$\nabla f$$ for some function $$f$$, if we are at position $$(v_1, v_2, ..., v_n)$$, we can "slide down" $$f$$ by going in direction $$-\nabla f(v_1, v_2, ..., v_n)$$.

With our example of $$f(x, y)=x^2 + y^2$$, the unit vectors are $$e_1=(1, 0)$$ and $$e_2=(0, 1)$$, because $$v_1=x$$ and $$v_2=y$$, and those vectors point in the direction of the $$x$$ and $$y$$ axes. Thus, $$\nabla f(x, y) = 2x (1, 0) + 2y(0, 1)$$.

Now, to "slide down" our function $$f$$, let's say we are at a point $$(-2, 4)$$. Then we would need to move in direction $$-\nabla f(-2, -4)= -(2 \cdot -2 \cdot (1, 0) + 2 \cdot 4 \cdot (0, 1)) = -((-4, 0) + (0, 8))=(4, -8)$$.

The magnitude of this vector will give us how steep the hill is (higher values means the hill is steeper). In this case, we have $$\sqrt{4^2+(-8)^2}\approx 8.944$$.

The Hadamard Product of two matrices $$A, B \in R^{n\times m}$$, is just like matrix addition, except instead of adding the matrices element-wise, we multiply them element-wise.

Formally, while matrix addition is $$A + B = C$$, where $$C \in R^{n \times m}$$ such that

$$C^i_j = A^i_j + B^i_j$$,

The Hadamard Product $$A \odot B = C$$, where $$C \in R^{n \times m}$$ such that

$$C^i_j = A^i_j \cdot B^i_j$$

(most of this section is from Neilsen's book).

We have a set of training samples, $$(S, E)$$, where $$S_r$$ is a single input training sample, and $$E_r$$ is the expected output value of that training sample. We also have our neural network, composed of weights $$W$$, and biases $$B$$. $$r$$ is used to prevent confusion from the $$i$$, $$j$$, and $$k$$ used in the definition of a feedforward network.

Next, we define a cost function, $$C(W, B, S^r, E^r)$$ that takes in our neural network and a single training example, and outputs how good it did.

Normally what is used is quadratic cost, which is defined by

$$C(W, B, S^r, E^r) = 0.5\sum\limits_j (a^L_j - E^r_j)^2$$

where $$a^L$$ is the output to our neural network, given input sample $$S^r$$

Then we want to find $$\frac{\partial C}{\partial w^i_j}$$ and $$\frac{\partial C}{\partial b^i_j}$$ for each node in our feedforward neural network.

We can call this the gradient of $$C$$ at each neuron because we consider $$S^r$$ and $$E^r$$ as constants, since we can't change them when we are trying to learn. And this makes sense - we want to move in a direction relative to $$W$$ and $$B$$ that minimizes cost, and moving in the negative direction of the gradient with respect to $$W$$ and $$B$$ will do this.

To do this, we define $$\delta^i_j=\frac{\partial C}{\partial z^i_j}$$ as the error of neuron $$j$$ in layer $$i$$.

We start with computing $$a^L$$ by plugging $$S^r$$ into our neural network.

Then we compute the error of our output layer, $$\delta^L$$, via

$$\delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma^{ \prime}(z^L_j)$$.

Which can also be written as

$$\delta^L = \nabla_a C \odot \sigma^{ \prime}(z^L)$$.

Next, we find the error $$\delta^i$$ in terms of the error in the next layer $$\delta^{i+1}$$, via

$$\delta^i=((W^{i+1})^T \delta^{i+1}) \odot \sigma^{\prime}(z^i)$$

Now that we have the error of each node in our neural network, computing the gradient with respect to our weights and biases is easy:

$$\frac{\partial C}{\partial w^i_{jk}}=\delta^i_j a^{i-1}_k=\delta^i(a^{i-1})^T$$

$$\frac{\partial C}{\partial b^i_j} = \delta^i_j$$

Note that the equation for the error of the output layer is the only equation that's dependent on the cost function, so, regardless of the cost function, the last three equations are the same.

As an example, with quadratic cost, we get

$$\delta ^L = (a^L - E^r) \odot \sigma ^ {\prime}(z^L)$$

for the error of the output layer. and then this equation can be plugged into the second equation to get the error of the $$L-1^{\text{th}}$$ layer:

$$\delta^{L-1}=((W^{L})^T \delta^{L}) \odot \sigma^{\prime}(z^{L-1})$$ $$=((W^{L})^T ((a^L - E^r) \odot \sigma ^ {\prime}(z^L))) \odot \sigma^{\prime}(z^{L-1})$$

which we can repeat this process to find the error of any layer with respect to $$C$$, which then allows us to compute the gradient of any node's weights and bias with respect to $$C$$.

I could write up an explanation and proof of these equations if desired, though one can also find proofs of them here. I'd encourage anyone that is reading this to prove these themselves though, beginning with the definition $$\delta^i_j=\frac{\partial C}{\partial z^i_j}$$ and applying the chain rule liberally.

For some more examples, I made a list of some cost functions alongside their gradients here.

Now that we have these gradients, we need to use them learn. In the previous section, we found how to move to "slide down" the curve with respect to some point. In this case, because it's a gradient of some node with respect to weights and a bias of that node, our "coordinate" is the current weights and bias of that node. Since we've already found the gradients with respect to those coordinates, those values are already how much we need to change.

We don't want to slide down the slope at a very fast speed, otherwise we risk sliding past the minimum. To prevent this, we want some "step size" $$\eta$$.

Then, find the how much we should modify each weight and bias by, because we have already computed the gradient with respect to the current we have

$$\Delta w^i_{jk}= -\eta \frac{\partial C}{\partial w^i_{jk}}$$

$$\Delta b^i_j = -\eta \frac{\partial C}{\partial b^i_j}$$

Thus, our new weights and biases are

$$w^i_{jk} = w^i_{jk} + \Delta w^i_{jk}$$ $$b^i_j = b^i_j + \Delta b^i_j$$

Using this process on a neural network with only an input layer and an output layer is called the Delta Rule.

Now that we know how to perform backpropagation for a single sample, we need some way of using this process to "learn" our entire training set.

One option is simply performing backpropagation for each sample in our training data, one at a time. This is pretty inefficient though.

A better approach is Stochastic Gradient Descent. Instead of performing backpropagation for each sample, we pick a small random sample (called a batch) of our training set, then perform backpropagation for each sample in that batch. The hope is that by doing this, we capture the "intent" of the data set, without having to compute the gradient of every sample.

For example, if we had 1000 samples, we could pick a batch of size 50, then run backpropagation for each sample in this batch. The hope is that we were given a large enough training set that it represents the distribution of the actual data we are trying to learn well enough that picking a small random sample is sufficient to capture this information.

However, doing backpropagation for each training example in our mini-batch isn't ideal, because we can end up "wiggling around" where training samples modify weights and biases in such a way that they cancel each other out and prevent them from getting to the minimum we are trying to get to.

To prevent this, we want to go to the "average minimum," because the hope is that, on average, the samples' gradients are pointing down the slope. So, after choosing our batch randomly, we create a mini-batch which is a small random sample of our batch. Then, given a mini-batch with $$n$$ training samples, and only update the weights and biases after averaging the gradients of each sample in the mini-batch.

Formally, we do

$$\Delta w^{i}_{jk} = \frac{1}{n}\sum\limits_r \Delta w^{ri}_{jk}$$

and

$$\Delta b^{i}_{j} = \frac{1}{n}\sum\limits_r \Delta b^{ri}_{j}$$

where $$\Delta w^{ri}_{jk}$$ is the computed change in weight for sample $$r$$, and $$\Delta b^{ri}_{j}$$ is the computed change in bias for sample $$r$$.

Then, like before, we can update the weights and biases via:

$$w^i_{jk} = w^i_{jk} + \Delta w^{i}_{jk}$$ $$b^i_j = b^i_j + \Delta b^{i}_{j}$$

This gives us some flexibility in how we want to perform gradient descent. If we have a function we are trying to learn with lots of local minima, this "wiggling around" behavior is actually desirable, because it means that we're much less likely to get "stuck" in one local minima, and more likely to "jump out" of one local minima and hopefully fall in another that is closer to the global minima. Thus we want small mini-batches.

On the other hand, if we know that there are very few local minima, and generally gradient descent goes towards the global minima, we want larger mini-batches, because this "wiggling around" behavior will prevent us from going down the slope as fast as we would like. See here.

One option is to pick the largest mini-batch possible, considering the entire batch as one mini-batch. This is called Batch Gradient Descent, since we are simply averaging the gradients of the batch. This is almost never used in practice, however, because it is very inefficient.

I haven't dealt with Neural Networks for some years now, but I think you will find everything you need here:

Neural Networks - A Systematic Introduction, Chapter 7: The backpropagation algorithm

I apologize for not writing the direct answer here, but since I have to look up the details to remember (like you) and given that the answer without some backup may be even useless, I hope this is ok. However, if any questions remain, drop a comment and I'll see what I can do.

I'll address the main confusion first: there is no error of a hidden layer. There's only an error of the output. The backpropagation = "back" (chain rule of differentiation) + "propagation" (information travels between layers). I'll explain.

The backpropagation term comes from the following intuition: your inputs propagate through layers into the outputs. Once you got the outputs, you compare them to actual observations and compute errors and their cost. Then you, sort of, propagate these errors backwards all the way to the inputs, while attributing the change in loss function to each weight and bias in the layers. The backward direction stems from the chain rule in calculus.

Here's how the chain rule works. Suppose your inputs are $$x$$, and you have two layer network that ultimately outputs $$\hat y$$. The output of a hidden layer with one parameter $$\theta$$ is $$g(x|\theta)$$, and the NN output from the last layer is then $$\hat y=f(g(x|\theta)|\phi)$$, where $$\phi$$ a single parameter of the output layer. You compare the output $$\hat y$$ with observation $$y$$, then compute the loss $$L(y,\hat y)$$.

So, you want to decrease the loss $$L(y,\hat y(x|\theta,\phi))$$, and in order to do this you want to try to change parameters by $$\Delta\theta,\Delta\phi$$ somehow. One way to do it is to see how sensitive is the output to the parameter changes, and it's measured by the gradient. So you go and compute it: $$\frac{\partial L}{\partial \phi}=\frac{\partial f}{\partial\phi}\frac{\partial}{\partial f}L$$ $$\frac{\partial L}{\partial \theta}=\frac{\partial g}{\partial\theta}\frac{\partial f}{\partial g}\frac{\partial}{\partial f}L$$

You can see how we, sort of, move backwards while calculating the sensitivities (slopes) of hidden layer parameters (weights) $$\frac{\partial L}{\partial \theta}$$. If you have more layers then the expression for gradient becomes longer with each layer, adding derivatives from the left. This is why it's called backpropagation. I don't like the term, but it's what ML folks use.

Once you got the gradient, you can now use it to determine in which direction and by how much to change your parameters for the next iteration. As you can see there is NO loss or error of a hidden layer. All we get is the sensitivity of the loss (error) $$L(y,\hat y)$$ to the weights in hidden layers.

• First 2 sentences are often overlooked in literature about deep learning! I spent too long trying to backpropagate in all the wrong places, so this answer should save a future newbie some time. May 6, 2021 at 0:29