0
$\begingroup$

$\mathcal{L}$ is the loss function, $\mathcal{L} = y_i \text{log} \sigma(z) + (1-y_i) \text{log} (1-\sigma(z))$, where $z = \sum_i w_ix_i$, with $w_i$ representing the weights and $x_i$ the features. Typically, it is required to take a derivative of $\mathcal{L}$ with respect to $w_1$ or $w_2$. But I am required to provide derivative with respect to parameters which includes all weights.

Could anyone please recommend a solution or some tutorials? Thanks

$\endgroup$
2
  • $\begingroup$ What do you mean by providing derivative wrt parameters which includes all weights'?' $\endgroup$
    – gunes
    May 21, 2020 at 8:08
  • $\begingroup$ What is $\sigma(x)$? The sigmoid function? $\endgroup$
    – Tinu
    May 21, 2020 at 8:18

1 Answer 1

0
$\begingroup$

Since $\mathcal{L}(w_1, w_2, ..., w_n)$ is a scalar loss function depending on multiple weights $w_i$ (neglecting features $x$ and labels $y$ for a moment) taking the gradient with respect to all would give you a vector

$\nabla \mathcal{L} = [\frac{\partial \mathcal{L}}{\partial w_1}, ...,\frac{\partial \mathcal{L}}{\partial w_n}]$.

Now you can calculate each individual component with the following formula, using the sigmoid derivative $\sigma'(x) = \sigma(x)(1 - \sigma(x))$ and chain rule:

$\frac{\partial \mathcal{L}}{\partial w_i} = \frac{y_ix_i\sigma(z)(1 - \sigma(z))}{\sigma(z)} - \frac{(1 - y_i)x_i\sigma(z)(1-\sigma(z))}{1 - \sigma(z)}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.