10
$\begingroup$

I'm looking at the cross-entropy cost function found in this tutorial:

$$C = -\frac{1}{n} \sum_x [y \ln a+(1−y)\ln(1−a)]$$

What exactly are we summing over? It is, of course, over $x$, but $y$ and $a$ don't change with $x$. All of the $x$'s are inputs into the one $a$. $a$ is even defined in the paragraph above the equation as a function of the sum of all $w$'s and $x$'s.

Also, $n$ is defined as the number of inputs into this particular neuron, correct? It is worded as "the total number of items of training data".


Edit:

Am I correct in thinking that

$$C= -\frac{1}{n} \sum_x [y \ln a+(1−y)\ln(1−a)]$$

would be the cost function for the entire network, whereas

$$C = [y \ln a+(1−y)\ln(1−a)]$$

would be the cost for the individual neuron? Shouldn't the sum be over each output neuron?

$\endgroup$
14
$\begingroup$

Here's how I would express the cross-entropy loss: $$\mathcal{L}(X, Y) = -\frac{1}{n} \sum_{i=1}^n y^{(i)} \ln a(x^{(i)}) + \left(1 - y^{(i)}\right) \ln \left(1 - a(x^{(i)})\right) $$

Here, $X = \left\{x^{(1)},\dots,x^{(n)}\right\}$ is the set of input examples in the training dataset, and $Y=\left\{y^{(1)},\dots,y^{(n)} \right\}$ is the corresponding set of labels for those input examples. The $a(x)$ represents the output of the neural network given input $x$.

Each of the $y^{(i)}$ is either 0 or 1, and the output activation $a(x)$ is typically restricted to the open interval (0, 1) by using a logistic sigmoid. For example, for a one-layer network (which is equivalent to logistic regression), the activation would be given by $$a(x) = \frac{1}{1 + e^{-Wx-b}}$$ where $W$ is a weight matrix and $b$ is a bias vector. For multiple layers, you can expand the activation function to something like $$a(x) = \frac{1}{1 + e^{-Wz(x)-b}} \\ z(x) = \frac{1}{1 + e^{-Vx-c}}$$ where $V$ and $c$ are the weight matrix and bias for the first layer, and $z(x)$ is the activation of the hidden layer in the network.

I've used the (i) superscript to denote examples because I found it to be quite effective in Andrew Ng's machine learning course; sometimes people express examples as columns or rows in a matrix, but the idea remains the same.

$\endgroup$
  • $\begingroup$ Thanks! So this would give us a single number for our error for the whole network, over all of our samples. For back propagation I need to find the partial derivative of this function wrt the weight matrix in the final layer. How would I do that? $\endgroup$ – Adam12344 Aug 19 '15 at 2:30
  • $\begingroup$ Doing backprop is a whole separate can of worms! The page you linked to has a description of computing derivatives etc. and there are many questions about backprop on stackoverflow and this site. Try looking around a bit and then posting a separate question specifically about backprop. $\endgroup$ – lmjohns3 Aug 19 '15 at 2:40
  • $\begingroup$ This might be useful for you in understanding backprop it goes through the back prop with a four layer neural network with a cross entropy loss in gory detail :) cookedsashimi.wordpress.com/2017/05/06/… $\endgroup$ – YellowPillow Jul 29 '17 at 10:10
5
$\begingroup$

What exactly are we summing over?

The tutorial is actually pretty explicit:

... $n$ is the total number of items of training data, the sum is over all training inputs...

The original single neuron cost function given in the tutorial (Eqn. 57) also has an $x$ subscript under the $\Sigma$ which is supposed to hint at this. For the single neuron case there's nothing else to sum over besides training examples, since we already summed over all the input weights when computing $a$:

$$ a = \sum_{j} w_jx_j. $$


Later on in the same tutorial, Nielsen gives an expression for the cost function for a multi-layer, multi-neuron network (Eqn. 63):

$$ C = -\frac{1}{n}\sum_{x}\sum_{j}[ y_j \ln a^{L}_{j} + (1 - y_j) \ln (1 - a^{L}_{j})]. $$

In this case the sum runs over both training examples ($x$'s) and individual neurons in the output layer ($j$'s).

$\endgroup$
  • $\begingroup$ Thanks for the insight, one question: the latter you defined is not the categorical cross entropy, right? $\endgroup$ – Tommaso Guerrini Feb 9 '17 at 9:54
  • $\begingroup$ He also said in his tutorial that "y can sometimes take values intermediate between 0 and 1" but the function he gave is all on y and there was no activation input. How could we implement intermediate values in the st function ? $\endgroup$ – Feras Feb 15 '17 at 17:56
  • $\begingroup$ In Nielsen's tutorial, which shows a single-layer perceptron, a = \sigma( \sum_{j} w_j x_j) because you have sigmoid activation function for your output layer, not a = \sum_{j} w_j x_j $\endgroup$ – ARAT Dec 13 '17 at 2:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.