I am having difficulty in understanding the cost function equation for neural nets described in andrew ng's coursera class. The equation is as follows :-

enter image description here

I am having difficulty in interpreting the regularization term. What does the 3 summations signify. I hope someone can break up the problem and help me understand the equation. Thanks in advance!


2 Answers 2


The first term looks a normal cross-entropy loss function.

The second term is a regularization term. It increases the cost function to "punish" large values for weights. It's called L2 weight-decay (L2 means it's using the square value, L1 means absolute value). This is done to prevent overfitting, because it makes it more difficult for the network to learn noise from input samples.

Now for the formula: (note that the formula is for a whole batch of m input samples).

The summation symbol is just another way of writing a for-loop so what the three summations means is this: for each layer l and for every pair of connecting units i and j, take the square of the weight from j to i, then add everything.

The lambda parameter is just used to control how much regularization you want; a large lambda means you want you want your network to be very regularized, a small lambda means you just want a little bit of regularization.

  • $\begingroup$ What exactly does i and j represent? And why is i looping till sl and j looping till (sl + 1)? $\endgroup$ Jun 26, 2017 at 3:37
  • $\begingroup$ i and j are hidden units. l is the layer index. From zero to (number of layers - 1). I think s_l is the number of units on layer l. $\endgroup$
    – Felipe
    Jun 26, 2017 at 3:41
  • $\begingroup$ To be more precise , i want to know why is i looping till s_l and j till (s_l + 1) ? There must be a reason i guess. $\endgroup$ Jun 26, 2017 at 3:45
  • $\begingroup$ Because you want to add all weights from layer l to layer l + 1, so you need s_l times s_l +1 weights, because it's a fully connected layer. $\endgroup$
    – Felipe
    Jun 26, 2017 at 3:49
  • $\begingroup$ Sorry , but I still didn't quite get it. What does a "fully connected layer" mean? Also , does i represent the previous layer and j the current hidden layer or vice- versa. I'm a relatively new in machine learning , so please try to explain in layman terms :) $\endgroup$ Jun 26, 2017 at 3:56

I am a bit late, but better late than never (at least for instructional purposes). Here is an example of a nn setup:

Neural Network setup

Here is the regularization term:

Regularization term

See how a nn theta matrix is arrayed:

Weights matrix

You can see that in any theta, the superscript "l" is the layer of "causing" units, and the two subscripts identify the activation unit in "l+1" and the "causing" unit in "l", respectively. Therefore, in the nn setup above, for example:

enter image description here

this is the theta that maps from unit x(1) in layer 1 to activation unit a(2) in layer 2. As you can see the second subscript is associated to the "causing" layer (l) and therefore its loop goes from 1 to s(1) (which is the last unit in layer 1 in the above setup, and is called s_subscript_"l" in the J function). The first subscript is associated to the activation unit in layer l+1 (the "caused" layer) and therefore its loop goes from 1 to s(2) (which is the last unit in layer 2 in the above setup, and is called s_subscript_"l+1" in the J function).

With that I hope the "i" and "j" loops get more intuitive now.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.