According to Andrew Ng's Coursera Machine Learning course (notes here and here), a training algorithm for neural nets is this:
- $\delta^{(L)} = a^{(L)} - y$
- $\text{for }l = L-1, ..., 2\text{ do}$
- $\quad \delta^{(l)} = (\Theta^{(l)})^T \delta^{(l + 1)} .* g'(z^{(l)})$
- $\text{end}$
Then, for computing the actual gradients:
- $\text{Set }\Delta^{(l)} = 0,\text{ for }l = 1, ..., L-1$
- $\text{for }i = 1, ..., m\text{ do}$
- $\quad \text{Set }a^{(1)} = x^{(i)}$
- $\quad \text{Compute }a^{(l)}\text{ for }l=2,...,L \text{ using forward propagation (Algorithm 1)}$
- $\quad \text{Using }y^{(i)}, \text{compute } \delta^{(l)} \text{ for } l = L, ..., 2 \\ \qquad\text{ using the backpropagation algorithm (Algorithm 2)}$
- $\quad \Delta^{(l)} = \Delta^{(l)} + \delta^{(l+1)}(a^{(l)})^T \text{ for }l=L-1, ..., 1$
- $\text{end}$
However, this seems to contradict this (page 7, taken from Mitchell's book I believe, but also in various other tutorials: here and here for example), where the output layer delta is defined as:
$${\delta^{(L)} = (a^{(L)} - y}).*(a^{(L)}.*(1-a^{(L)}))$$
So the derivative of the output layer outputs is also factored in. In the above, $a$ always represents the output of a layer and $z$ is the "raw computation" of a layer, before the sigmoid function is applied to it. $.*$ is element-wise multiplication.
According to what I learned and understand, you should factor in the derivative as well. According to my experiments, things work a little better if you do, but I haven't been able to test too much.
Which is the correct way? Or are they equivalent but I misunderstood something about the first notation (and therefore also implemented it wrong)? Or it doesn't matter and I'll just need proper hyperparameters for each to obtain similar results?
Here is some code that shows what I mean:
X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1] ])
y = np.array([[0,1,1,0]]).T
syn0 = 2*np.random.random((3,4)) - 1
syn1 = 2*np.random.random((4,1)) - 1
for j in xrange(60000):
l1 = 1/(1+np.exp(-(np.dot(X,syn0))))
l2 = 1/(1+np.exp(-(np.dot(l1,syn1))))
l2_delta = (y - l2)*(l2*(1-l2)) #######################
l1_delta = l2_delta.dot(syn1.T) * (l1 * (1-l1))
syn1 += l1.T.dot(l2_delta)
syn0 += X.T.dot(l1_delta)
According to my understanding, Andrew Ng's version suggests replacing the line marked with hashtags with just this:
l2_delta = y - l2
Also according to my understanding, doing that would be wrong.