The role of the regularization parameter while optimizing a function

Question

I'm reading a paper where the aim is to optimize the following function:

$h()$ is a loss function and $\lambda$ is a regularization parameter. The idea in $h()$ is that whenever $p_l-p_d > 0$ then $h()$ will give a penalty.

Now the basic way to optimize this function is to compute the gradient of the function and then use gradient descent. Then one just keeps updating the gradients values until the termination condition becomes valid in gradient descent.

But then my question is: doesn't the role of the regularization parameter vanishes here? Because when you compute the gradient of the function, then the regularization parameter will disappear from the gradient equation.

Here's the partial derivative of the function from the paper:

So what I don't understand is only the role of the regularization parameter. I hope if you can clarify it.

Answer 1

The regularization parameter cannot disappear, because it multiplies the sum:

$$\frac {\partial F}{\partial w} = 2w + \frac {\partial}{\partial w}\left(\lambda \sum_{l,d}h(p_l-p_d)\right) = 2w + \lambda \frac {\partial}{\partial w}\left(\sum_{i,d}h(p_l-p_d)\right)$$

$$=2w + \lambda \left(\sum_{i,d}\frac {\partial h(p_l-p_d)}{\partial w}\right)$$

e.t.c

Later on on the paper (section 5.1 -(D)) the authors say that they found that "setting $\lambda=1$ gives best performance". I believe they had that in mind and were led to write their initial equations mistakenly.