I've implemented 3 supervised training algorithms: rprop, online- and batch backprop with momentum. I have the simple XOR test, and I measured how many times they converge out of N iterations. My results:
error tresshold is 5% momentum is 0.7 rprop init. updates are 0.1 learning rate is 1.0 maximum epoch counter is 10000 networks are constructed with random weights [-1, 1). a xor network has 2 input + 1 input bias, 3 hidden + 1 bias and 1 output neurons, sigmoid activation function everywhere RMS error ------------------------------- Online: 96% success rate (global error is under error tresshold) 1100 iterations averaged (it is only accumulated on successful trainings, since failed runs have max iterations) Batch: 99% success rate 770 iterations average Resilient: 80% success rate 40 iterations average
The online learning seems to be the slowest. Resilient is pretty fast, but it often converges to 0.35 (maybe it is a local minima of this network, however I cannot find any proof about it). Batch is precise, and faster than online. Online even yields 0.011 error in 10k iterations, which is very slow.
Also, decresing the number of hidden units to 2 makes the convergence fail more often, but decreses the number of iterations on successful attempts.
Is there any free paper about the 3 hidden-layer XOR network (2 hidden-layer ffnn converges to this too), and if it has a local minima (at 0.35), or it is just my implementation problem? (I've found publications, but they are not free).