Problem in understanding Regularized Cost Function for neural neworks I am having difficulty in understanding the cost function  equation for neural nets described in andrew ng's coursera class. The equation is as follows :-

I am having difficulty in interpreting the regularization term. What does the 3 summations signify. I hope someone can break up the problem and help me understand the equation. Thanks in advance!
 A: The first term looks a normal cross-entropy loss function.
The second term is a regularization term. It increases the cost function to "punish" large values for weights. It's called L2 weight-decay (L2 means it's using the square value, L1 means absolute value). This is done to prevent overfitting, because it makes it more difficult for the network to learn noise from input samples.
Now for the formula: (note that the formula is for a whole batch of m input samples).
The summation symbol is just another way of writing a for-loop so what the three summations means is this: for each layer l and for every pair of connecting units i and j, take the square of the weight from j to i, then add everything.
The lambda parameter is just used to control how much regularization you want; a large lambda means you want you want your network to be very regularized, a small lambda means you just want a little bit of regularization.
A: I am a bit late, but better late than never (at least for instructional purposes). Here is an example of a nn setup:

Here is the regularization term:

See how a nn theta matrix is arrayed:

You can see that in any theta, the superscript "l" is the layer of "causing" units, and the two subscripts identify the activation unit in "l+1" and the "causing" unit in "l", respectively. Therefore, in the nn setup above, for example:

this is the theta that maps from unit x(1) in layer 1 to activation unit a(2) in layer 2. As you can see the second subscript is associated to the "causing" layer (l) and therefore its loop goes from 1 to s(1) (which is the last unit in layer 1 in the above setup, and is called s_subscript_"l" in the J function). The first subscript is associated to the activation unit in layer l+1 (the "caused" layer) and therefore its loop goes from 1 to s(2) (which is the last unit in layer 2 in the above setup, and is called s_subscript_"l+1" in the J function).
With that I hope the "i" and "j" loops get more intuitive now.
