I'm trying to solve an exercise. I'm asked to train a 2-layer neural network, given a small dataset. Below is a picture I drew to understand the architecture and derive the formula for backpropagation.

enter image description here

In this case I just have two input features and a bias unit. Moreover, $h = w_1x_1 + w_2x_2 + w_3$, $\hat{y} = \sigma(h)$, which is the output, and $E = -y ln \hat{y} - (1-y) ln (1-\hat{y})$, with $y$ being the real label.

If I want to compute the partial derivative of the error wrt $w_1$, I would do the following: $$ \frac{\partial}{\partial w_1} E = \frac{\partial}{\partial \hat{y}} \frac{\partial}{\partial h} \hat{y} \frac{\partial}{\partial w_1} h. $$

This should simply be $-(y - \hat{y}) x_1$. Nonetheless, in the skeleton I was given, the update for the weights reads:

$$ W = W + \frac{\alpha}{m} (y-\hat{y}) \sigma ' (x) x, $$

where $m$ is just the number of examples. I do not understand where that $\sigma ' (x)$ comes from.

Moreover, the update for the weights is done after each epoch, and not after each training example, as I read in some textbooks. Is this another way to perform learning?

  • $\begingroup$ You update the weights after each iteration. Epochs are just a programming tool and don't really have any statistical meaning. $\endgroup$ – generic_user Nov 17 '18 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.