I'm reading a paper where the aim is to optimize the following function:

enter image description here

$h()$ is a loss function and $\lambda$ is a regularization parameter. The idea in $h()$ is that whenever $p_l-p_d > 0$ then $h()$ will give a penalty.

Now the basic way to optimize this function is to compute the gradient of the function and then use gradient descent. Then one just keeps updating the gradients values until the termination condition becomes valid in gradient descent.

But then my question is: doesn't the role of the regularization parameter vanishes here? Because when you compute the gradient of the function, then the regularization parameter will disappear from the gradient equation.

Here's the partial derivative of the function from the paper:

enter image description here

So what I don't understand is only the role of the regularization parameter. I hope if you can clarify it.


The regularization parameter cannot disappear, because it multiplies the sum:

$$\frac {\partial F}{\partial w} = 2w + \frac {\partial}{\partial w}\left(\lambda \sum_{l,d}h(p_l-p_d)\right) = 2w + \lambda \frac {\partial}{\partial w}\left(\sum_{i,d}h(p_l-p_d)\right)$$

$$=2w + \lambda \left(\sum_{i,d}\frac {\partial h(p_l-p_d)}{\partial w}\right)$$


Later on on the paper (section 5.1 -(D)) the authors say that they found that "setting $\lambda=1$ gives best performance". I believe they had that in mind and were led to write their initial equations mistakenly.

  • $\begingroup$ ok so it's just a mistake :) $\endgroup$
    – Jack Twain
    May 18 '14 at 11:20
  • $\begingroup$ Yes, just an oversight. $\endgroup$ May 18 '14 at 11:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.