Batch gradient descent in Perceptron linear classifier

Question

I'm learning about batch gradient descent for the Perceptron linear classifier and I'm confused about the update rule. On Wikipedia, it says that the update rule for batch gradient descent is $w := w - \alpha \sum_{i = 1}^{n} \nabla Q_i(w)$ where $\alpha$ is the learning rate.

Why is the gradient a sum and not an average of the gradients of each misclassified sample? I tried implementing the batch gradient descent update rule above and it seemed to make the error worse since the weights would update by a huge amount at each iteration. Instead, I tried $w := w - \alpha \frac{1}{n} \sum_{i = 1}^{n} \nabla Q_i(w)$ and the results were much better. Am I understanding the update rule incorrectly?

Answer 1

The cost function uses the sum of the errors instead of the average because the sum ensures that the update takes a step in the direction of the steepest decrease of the cost function. Using the average of the errors instead of the sum would result in what amounts to a less effective step and possibly in the wrong direction.

Think of a single training example where the error is large due to some weight. In this case, using the average of the errors over the whole set would essentially smooth out the cost of using this specific weight.

It seems like you have a good understanding of the update rule. However, the problem you are describing seems to be related to the scale of the error. Using a smaller learning rate should help to take care of this.