Intuition behind output neuron error signal in backpropagation

Question

I have a potentially obvious question about neural network error backpropagation.

In Andrej Karpathy's blog post on neural networks, he goes through an example of backpropagation.

When calculating the error gradient at the output of the network (which is then backpropagated to get the derivative of the output error with respect to each weight), he does:

dscores = probs
dscores[range(num_examples),y] -= 1
dscores /= num_examples,

where probs is an array where each row represents an output class probability distribution for a given training observation, num_examples is the number of training observations, and y is the output class (which corresponds to an index of the dscores array.

Can anyone briefly explain why he divides each observation by the number of training examples? I'm struggling to understand this.

Answer 1

Can anyone briefly explain why he divides each observation by the number of training examples?

dscores is the gradient computed based on all samples of the mini-batch (or in the tutorial you pointed to, the entire training set): it is divided by num_examples in order not to depend on size of the mini-batch (i.e., the number of samples in the mini-batch). It is commonplace to do so, e.g. this way, you don't have to change step_size if num_examples changes.

Answer 2

A given input in a neural network pass in forward direction to compute the output with a given set of initial values of parameters. And error signal pass in backward direction to update the corresponding values of parameters in order to minimize the total error.