3
$\begingroup$

I'm using the cross-entropy cost function for backpropagation in a neutral network as it is discussed in neuralnetworksanddeeplearning.com. I got help on the cost function here: Cross-entropy cost function in neural network

I'm confused on: $\frac{\partial C}{\partial w_j}= \frac1n \sum x_j(\sigma(z)−y)$

I'm not sure what $w_j$, $x_j$, $\sigma(z)$ are.

Is $w_j$ the matrix for the final layer, $x_j$ is input vector, $\sigma(z)$ is output vector? That doesn't really make sense, but I'm not sure what it is.

$\endgroup$
  • $\begingroup$ Anybody have any idea? $\endgroup$ – Adam12344 Aug 20 '15 at 2:04
  • 1
    $\begingroup$ This may be useful for you it works through the backpropagation algorithm in a 4 layer neural network with cross entropy loss: cookedsashimi.wordpress.com/2017/05/06/… $\endgroup$ – YellowPillow May 8 '17 at 1:08
2
$\begingroup$

Let us assume that the activation function is a logistic regression denoted as $\sigma()$.

The idea behind cross-entropy (CE) is to optimise the weights $W = [w_1, w_2,...,w_j,...w_k]$ to maximise the log probability - or to minimise the negative log probability.

Here, you are willing to obtain each neuron's derivative of the cost $C^n$ with respect to each of the layers in $W$. Thus, you write $\frac{\partial C}{\partial w_j}$, where $C = [C^1, C^2,...,C^n,...,C^m]$. After some math, which I'll skip here but you can read more about it (in case you're interested here (slide 18 proves useful) and here):

This results in $\frac{1}{n} \sum x_j(\sigma(z)−y)$, where $n$ is the size of your set.

Here, $z=WX+b$, where $X = [x_{11} \ x_{12}...x_{1j}...x_{1k};\quad ....;\quad x_{n1} \ x_{n2}...x_{nj}...x_{nk}]$ ($X$ is an $n$ by $k$ matrix) and $x_{11}..x_{1k}$ are the features you would have per entry, $W$ are the weights as defined above and $b$ is the bias.

In classification, you would like to use this linear dependency of $z$. However, you would want to run it through a non-linear function such as a sigmoid, hereby defined by $\sigma()$ (you can see a proof and read more about it here). $y$ represents the targeted output.

So $w_j$ is the j-th weight of the vector above; $x_j$ is the j-th input vector of an entry, $\sigma(z)$ is the sigmoid applied to the $WX+b$ linear function.

Hope that makes sense.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.