# How to compute the derivative of the total loss wrt external trainable parameters?

I was just curious how external trainable parameters are updated. The challenge is to compute the derivative, the rest is handled by the optimiser.

I assumed a simple DNN as follows: $$\hat{y} =\sigma(W_2 \sigma(W_1 X + b_1) + b_2)$$

where the symbols have their usual meaning. The full forward pass is as follows:

$$a^{(0)} = X$$ $$a^{(i)} = \sigma(z^{(i)})$$ $$z^{(i)} = W_i a^{(i-1)} + b_i$$ $$\hat{y} = a^{(L)}$$

The derivative of the loss wrt the inputs would be: $$\frac{\partial J}{\partial X} = \frac{\partial J}{\partial a^{(2)}} \frac{\partial a^{(2)}}{\partial z^{(2)}} \frac{\partial z^{(2)}}{\partial a^{(1)}} \frac{\partial a^{(1)}}{\partial z^{(1)}} \frac{\partial z^{(1)}}{\partial X}$$

This is what I know.

Now, let us look at the multitask loss function as follows:

$$J = \lambda_1 Y_1 + \lambda_2 Y_2$$

where $$Y_1$$ and $$Y_2$$ are outputs of the DNN with external trainable parameters $$\lambda_1$$ and $$\lambda_1$$.

The weight updation is as usual. And I think the updation of external trainable parameters is as follows: $$\frac{\partial{J}}{\partial{\lambda_1}}=Y_1$$ $$\frac{\partial{J}}{\partial{\lambda_2}}=Y_2$$

Is that correct? Is it that simple?

Makes no sense to have those $$\lambda$$ as trainable, because the optimizer will just use them to minimize the loss.

For example, take $$Y_1 = \text{MSE}$$ and $$Y_2 = \text{MAE}$$, then the loss will always be $$\ge 0$$... minimizing a loss means finding a minimum... if you use any optimizer on that loss, you will see that he will set the $$\lambda$$ to $$-\infty$$ an will maximize the MSE and the MAE... in other words, you won't get any good result out of this

OT:

• In your forward pass, you are missing an affine transformation
• your gradient is not differentiating with respect to $$W_1$$
• Ok, it was my mistake. I tried to oversimplify the loss function. For example, in this case the $\lambda$ won't go to negative infinity. I was just curious how this derivative thing actually works behind the scene. forums.fast.ai/t/loss-function-with-learnable-parameters/65301/… Jul 11, 2022 at 12:06
• @PrakharSharma yes, you can just differentiate like you did in the question, however you usually has to impose a prior over those trainable hyperparameter, like they did in the question you linked with the last term Jul 11, 2022 at 12:16