I'm trying to train a neural net with

  • 1 hidden layer (RELU)

  • softmax output layer

  • cross-entropy loss

  • stochastic gradient descent

My implementation seems to work fine when I don't use any hidden layers. I am using the following to update a weight from (input) layer $i$ to (output) layer $j$

$$\delta_j = t_j - y_j$$ $$\Delta{w_{(i,j)}} = \alpha \delta_j y_i$$

where $t$ is the (hot-encoded) target distribution and $y$ is the activation.

The confusion arises when I introduce the hidden RELU layer.

My question is how to update the weights between hidden layers? I tried using the following formula but it seems to be wrong (It works fine for square-loss but I'm confused how to modify it to work with cross-entropy)

$$\delta_j = (\sum_k \delta_k w_{(j,k)}) f'(y_j)$$ $$\Delta{w_{(i,j)}} = \alpha \delta_j y_i$$

where $k$ is the forward layer and $f$ is the RELU.