# Backpropagation with Cross-entropy Cost Function

I'm using the cross-entropy cost function for backpropagation in a neutral network as it is discussed in neuralnetworksanddeeplearning.com. I got help on the cost function here: Cross-entropy cost function in neural network

I'm confused on: $\frac{\partial C}{\partial w_j}= \frac1n \sum x_j(\sigma(z)−y)$

I'm not sure what $w_j$, $x_j$, $\sigma(z)$ are.

Is $w_j$ the matrix for the final layer, $x_j$ is input vector, $\sigma(z)$ is output vector? That doesn't really make sense, but I'm not sure what it is.

• Anybody have any idea? – Adam12344 Aug 20 '15 at 2:04
• This may be useful for you it works through the backpropagation algorithm in a 4 layer neural network with cross entropy loss: cookedsashimi.wordpress.com/2017/05/06/… – YellowPillow May 8 '17 at 1:08

Let us assume that the activation function is a logistic regression denoted as $\sigma()$.

The idea behind cross-entropy (CE) is to optimise the weights $W = [w_1, w_2,...,w_j,...w_k]$ to maximise the log probability - or to minimise the negative log probability.

Here, you are willing to obtain each neuron's derivative of the cost $C^n$ with respect to each of the layers in $W$. Thus, you write $\frac{\partial C}{\partial w_j}$, where $C = [C^1, C^2,...,C^n,...,C^m]$. After some math, which I'll skip here but you can read more about it (in case you're interested here (slide 18 proves useful) and here):

This results in $\frac{1}{n} \sum x_j(\sigma(z)−y)$, where $n$ is the size of your set.

Here, $z=WX+b$, where $X = [x_{11} \ x_{12}...x_{1j}...x_{1k};\quad ....;\quad x_{n1} \ x_{n2}...x_{nj}...x_{nk}]$ ($X$ is an $n$ by $k$ matrix) and $x_{11}..x_{1k}$ are the features you would have per entry, $W$ are the weights as defined above and $b$ is the bias.

In classification, you would like to use this linear dependency of $z$. However, you would want to run it through a non-linear function such as a sigmoid, hereby defined by $\sigma()$ (you can see a proof and read more about it here). $y$ represents the targeted output.

So $w_j$ is the j-th weight of the vector above; $x_j$ is the j-th input vector of an entry, $\sigma(z)$ is the sigmoid applied to the $WX+b$ linear function.

Hope that makes sense.