In a single layered neural network with a sigmoid function (to make it easy to understand) the cost function is

$E_j = \frac{1}{2} \sum_{k=1}^{K}(\text{target}_{jk} - \text{observed}_{jk})^2 + a\sum_{i=1}^{I}w_{ij}^2.$

So this is a two part problem and I know how to derive the first half at both the hidden and output layers. But, for the second half, I am not sure.

Any advice or help would be great. Thank you.

  • $\begingroup$ What is the question? What are the two "halves" you refer to? $\endgroup$
    – whuber
    Feb 14 '13 at 21:57
  • $\begingroup$ The two halves of the equation The left part of the sum Vs. the right side. The a(sum((w_ij^2) for i=1,2,..I) where a is a regulatory factor and w is the weight of NN fro i to j $\endgroup$ Feb 15 '13 at 2:52

It's the same formula but with different notations, if there is something that you don't understand, tell me

$J(\Theta ) = \frac{1}{2m}[\sum_{i=1}^{m} ( h_{\Theta }(x^{(i)}) - y^{(i)})^{2} + \lambda \sum_{j=1}^{n}\Theta_{j}^{2}]$

The derivative w.r.t $\Theta$

$\frac{\partial }{\partial \Theta_{j}} = \frac{1}{m}[\sum_{i=1}^{m} ( h_{\Theta }(x^{(i)}) - y^{(i)}))x_{j}^{(i)} + \lambda \Theta_{j}]$

This will give you the update for the weight $j$, that you use with gradient descent: $\Theta_{j} \leftarrow \Theta_{j} - \eta \frac{\partial }{\partial \Theta_{j}}$ And you do that for all your weights from $j = (1, 2, ..., n)$

  • $\begingroup$ Thank you for the response. But first isn't d/dx(f(x) + g(x)) = d/dxf(x) + d/dxg(x)? and also the lambda here is actually a number right? so d/dx n where n is a number = 0 so that completely knocks out the right side of the equation? $\endgroup$ Feb 18 '13 at 6:55
  • $\begingroup$ Yes d/dx(f(x) + g(x)) = d/dx f(x) + d/dx g(x) Yes d/dx n where n doesn't depend on the weight. No it doesn't knock out the right side, you have $\Theta_{j}$, you can't just delete it. $\endgroup$
    – ThiS
    Feb 18 '13 at 10:50
  • $\begingroup$ Yes ofcourse that was dumb of me d/d_theta(lambdatheta_j) is lamdad/d_theta(theda_j). How about in a back propagation though, we are not doing d/d_theta_j anymore. let say z_j such that z_j = x_j_iw_j_i. What would the right hand side (d/dz_j(lamdatheta^2)) look like? I was thinking 2*lamda(d/dz_j(theta_j)) but I cannot figure out how they are related and thus derive that. $\endgroup$ Feb 18 '13 at 15:02
  • $\begingroup$ What you are doing is computing the derivative for each weight: d/d_theta_1, d/d_theta_2, ..., d/d_theta_n for the update. For example, when updating d/d_theta_1, you have before the derivative the right side = lambdasum_for_j=1_to_n(theta_j)^2 If you expand your sum you have lambda*(theta_1^2+theta_2^2+...+theta_n^2). When you derivate, if you take the derivative w.r.t theta_1, then theta_2^2 is equal to 0 for example, so you just have to derivate theta_1^2, so it will be 2*lambdatheta_1. Hope you get it know $\endgroup$
    – ThiS
    Feb 18 '13 at 15:46
  • $\begingroup$ Thank you for your reply. Yes you are right. However your answer did not apply to the question. As this was not a derivation question, but a neural network problem and you do not do d/d-theta here. You changed my equation to a function, were as my question were matrices in a ANN. thx. $\endgroup$ Feb 19 '13 at 22:02

The regularization parameter is a form of Tikhonov's regulraization term (which is derived from the point of view of adding a preference to the particular solution when looking for the minimum of residuals)- as there are many ways of regularizing the neural network error function. You are simply trying to reduce the risk of overfitting by "punishing" your network for having big weights values. This $$ a\sum_{i,j} w_{ij}^2 $$ term is simply a weighted square of the weights norm $$ a \left \| w \right \|^2 = \left \| (\sqrt{a}I) w \right \|^2 $$ where $a$ is a constant used for balancing the networks fitting to the data ("left half") and its complexity ("right half"). The notion of network complexity is a complex issue on itself, this is why there have been many ideas for a regularization terms. In general, most of the currently used are methods of Tikhonov's regularization, but these are not the only ones.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.