1
$\begingroup$

I will classify using a neural network algorithm. I use 2 output, Y1=1 (positive) and Y2=0 (negative). The architecture is as follows:

enter image description here

loss that I use is binary cross entropy with the following formula:

enter image description here

From the loss, the error gradient with respect to weight v is:

enter image description here

based on the architecture that I demonstrated, is it correct to calculate the error gradient with respect to v11 like this?

enter image description here

$\endgroup$
6
  • $\begingroup$ where is the architecture? $\endgroup$
    – gunes
    Commented Jan 9, 2023 at 7:51
  • $\begingroup$ @gunes oow sorry, I just added $\endgroup$
    – Andryan
    Commented Jan 9, 2023 at 7:58
  • $\begingroup$ what are the activations? $\endgroup$
    – gunes
    Commented Jan 9, 2023 at 8:52
  • $\begingroup$ It's sigmoid biner $\endgroup$
    – Andryan
    Commented Jan 9, 2023 at 10:04
  • $\begingroup$ For a binary classification problem (two mutually exclusive classes), either you only use 1 output neuron, or you have to use an activation on the output $y_1, y_2$. But you have 2 units, and I don't see an activation function for $y_i$. The problem is that, as it's written, you have $\log y_i$, but in the absence of an activation, it's possible (even likely) to have $Wz + b \le 0$, which means that $\log y_i$ is not a real number. If you mean $y_i=\sigma(W_kz + b_k)$, then it is not necessarily true that $y_1 + y_2 = 1$, so the outputs are not probabilities of mutually exclusive events. $\endgroup$
    – Sycorax
    Commented Jan 9, 2023 at 14:21

1 Answer 1

0
$\begingroup$

The substitution you made is correct. The formula you use is also correct assuming sigmoid activation at each layer:

According to the architecture, the forward and backward propagation equations are: $$\begin{align}Z_j&=\sigma\left(\sum_{i=1}^{n_{input}} X_iV_{ij}+b_j\right)\rightarrow \frac{\partial Z_j}{\partial V_{ij}}=X_iZ_j(1-Z_j) \\Y_k&=\sigma\left(\sum_{j=1}^{n_{hidden}} Z_jW_{jk}+b_k\right)\rightarrow \frac{\partial Y_k}{\partial Z_j}=Y_k(1-Y_k)W_{jk} \\\frac{\partial Y_k}{\partial V_{ij}}&=\sum_{j'=1}^{n_{hidden}}\frac{\partial Y_k}{\partial Z_{j'}}\frac{\partial Z_{j'}}{\partial V_{ij}}=\frac{\partial Y_k}{\partial Z_{j}}\frac{\partial Z_j}{\partial V_{ij}}=Y_k(1-Y_k)W_{jk}X_iZ_j(1-Z_j) \\E&=-\sum_{k=1}^{n_{out}}Q_k\log Y_k+(1-Q_{k})\log(1-Y_k)\rightarrow \frac{\partial E}{\partial Y_k}=\frac{Y_k-Q_k}{Y_k(1-Y_k)}\end{align}$$

Using the chain rule, we have $$\begin{align}\frac{\partial E}{\partial V_{ij}}&=\sum_{k=1}^{n_{out}}\frac{\partial E}{\partial Y_k}\frac{\partial Y_k}{\partial V_{ij}}=\sum_{k=1}^{n_{out}}(Y_k-Q_k)W_{jk}X_iZ_j(1-Z_j)\end{align}$$

And, substituting $i=j=1$ here yields the following: $$\begin{align}\frac{\partial E}{\partial V_{11}}&=\sum_{k=1}^2(Y_k-Q_k)W_{1k}X_1Z_1(1-Z_1)\\&=(Y_1-Q_1)W_{11}X_1Z_1(1-Z_1)+(Y_2-Q_2)W_{12}X_1Z_1(1-Z_1)\end{align}$$

$\endgroup$
2
  • $\begingroup$ If we compute $y_i = \sigma(z w_i + b_i)$ as the ordinary sigmoid function, then it is not necessarily true that $y_1 + y_2 = 1$. Does that complicate this result, or the loss computation, at all? $\endgroup$
    – Sycorax
    Commented Jan 9, 2023 at 14:37
  • $\begingroup$ Haven't checked but it should a bit. Though the OP didn't use something like softmax layer (or one neuron) to ensure this. At the same time, the error is not in a multi-class cross-entropy form. But, as far as I understand, this is the architecture and the loss function, the formula given is not something the OP derived but only try to use. So, in this form, it holds. $\endgroup$
    – gunes
    Commented Jan 12, 2023 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.