I am having difficulty in understanding the cost function equation for neural nets described in andrew ng's coursera class. The equation is as follows :-

enter image description here

I am having difficulty in interpreting the regularization term. What does the 3 summations signify. I hope someone can break up the problem and help me understand the equation. Thanks in advance!


The first term looks a normal cross-entropy loss function.

The second term is a regularization term. It increases the cost function to "punish" large values for weights. It's called L2 weight-decay (L2 means it's using the square value, L1 means absolute value). This is done to prevent overfitting, because it makes it more difficult for the network to learn noise from input samples.

Now for the formula: (note that the formula is for a whole batch of m input samples).

The summation symbol is just another way of writing a for-loop so what the three summations means is this: for each layer l and for every pair of connecting units i and j, take the square of the weight from j to i, then add everything.

The lambda parameter is just used to control how much regularization you want; a large lambda means you want you want your network to be very regularized, a small lambda means you just want a little bit of regularization.

  • $\begingroup$ What exactly does i and j represent? And why is i looping till sl and j looping till (sl + 1)? $\endgroup$ – Anukarsh Singh Jun 26 '17 at 3:37
  • $\begingroup$ i and j are hidden units. l is the layer index. From zero to (number of layers - 1). I think s_l is the number of units on layer l. $\endgroup$ – Felipe Almeida Jun 26 '17 at 3:41
  • $\begingroup$ To be more precise , i want to know why is i looping till s_l and j till (s_l + 1) ? There must be a reason i guess. $\endgroup$ – Anukarsh Singh Jun 26 '17 at 3:45
  • $\begingroup$ Because you want to add all weights from layer l to layer l + 1, so you need s_l times s_l +1 weights, because it's a fully connected layer. $\endgroup$ – Felipe Almeida Jun 26 '17 at 3:49
  • $\begingroup$ Sorry , but I still didn't quite get it. What does a "fully connected layer" mean? Also , does i represent the previous layer and j the current hidden layer or vice- versa. I'm a relatively new in machine learning , so please try to explain in layman terms :) $\endgroup$ – Anukarsh Singh Jun 26 '17 at 3:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.