0
$\begingroup$

I was just curious how external trainable parameters are updated. The challenge is to compute the derivative, the rest is handled by the optimiser.

I assumed a simple DNN as follows: $$\hat{y} =\sigma(W_2 \sigma(W_1 X + b_1) + b_2)$$

where the symbols have their usual meaning. The full forward pass is as follows:

$$ a^{(0)} = X$$ $$a^{(i)} = \sigma(z^{(i)})$$ $$ z^{(i)} = W_i a^{(i-1)} + b_i$$ $$\hat{y} = a^{(L)} $$

The derivative of the loss wrt the inputs would be: $$ \frac{\partial J}{\partial X} = \frac{\partial J}{\partial a^{(2)}} \frac{\partial a^{(2)}}{\partial z^{(2)}} \frac{\partial z^{(2)}}{\partial a^{(1)}} \frac{\partial a^{(1)}}{\partial z^{(1)}} \frac{\partial z^{(1)}}{\partial X} $$

This is what I know.

Now, let us look at the multitask loss function as follows:

$$J = \lambda_1 Y_1 + \lambda_2 Y_2$$

where $Y_1$ and $Y_2$ are outputs of the DNN with external trainable parameters $\lambda_1$ and $\lambda_1$.

The weight updation is as usual. And I think the updation of external trainable parameters is as follows: $$\frac{\partial{J}}{\partial{\lambda_1}}=Y_1$$ $$\frac{\partial{J}}{\partial{\lambda_2}}=Y_2$$

Is that correct? Is it that simple?

$\endgroup$
0

1 Answer 1

0
$\begingroup$

Makes no sense to have those $\lambda$ as trainable, because the optimizer will just use them to minimize the loss.

For example, take $Y_1 = \text{MSE}$ and $Y_2 = \text{MAE}$, then the loss will always be $\ge 0$... minimizing a loss means finding a minimum... if you use any optimizer on that loss, you will see that he will set the $\lambda$ to $-\infty$ an will maximize the MSE and the MAE... in other words, you won't get any good result out of this

OT:

  • In your forward pass, you are missing an affine transformation
  • your gradient is not differentiating with respect to $W_1$
$\endgroup$
2
  • $\begingroup$ Ok, it was my mistake. I tried to oversimplify the loss function. For example, in this case the $\lambda$ won't go to negative infinity. I was just curious how this derivative thing actually works behind the scene. forums.fast.ai/t/loss-function-with-learnable-parameters/65301/… $\endgroup$ Jul 11, 2022 at 12:06
  • $\begingroup$ @PrakharSharma yes, you can just differentiate like you did in the question, however you usually has to impose a prior over those trainable hyperparameter, like they did in the question you linked with the last term $\endgroup$
    – Alberto
    Jul 11, 2022 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.