Dealing with a very basic neural language model: 3 words of context, vector size 100, one hidden layer size 200, vocabulary size 1000, predicting the next word with a softmax output layer.
Previously I have trained similar models with a stochastic (online) gradient descent - updating the parameters after each training example. This time, I wanted to try batch learning to speed up the matrix operations. However, the system is getting a big drop in accuracy (a rise in perplexity) going from batch size 1 to 2, and it gets even worse with bigger batches.
Training on 200K words for 10 epochs, the perplexities are as follows:
Batch size Perplexity 1 61.13 2 189.69 10 566.53
Batch sizes 1 and 2 converge to these values, batch size 10 actually diverges. Also tried halving the learning rate after each epoch - this gave 415.92 with batch size 10.
Initially, I was hoping that I just have a bug somewhere, but as far as I can tell that's not the case. I even created batches that are full of the same example, dividing the learning rate by batch size, and the system gave the same result as a stochastic training process.
I expected batch training to get a small increase in perplexity due to parameters being updated less frequently. But also thought that this would be offset by the benefit of averaging out parameter updates before applying them.
Has anyone compared batch vs stochastic training on a feedforward neural network LM? Does such a result make sense on this task? Is there perhaps something that should be handled differently when batch training? Or do you think I still have a sneaky bug somewhere? Many thanks for suggestions!