XOR backpropagation convergence

Question

I've implemented 3 supervised training algorithms: rprop, online- and batch backprop with momentum. I have the simple XOR test, and I measured how many times they converge out of N iterations. My results:

error tresshold is 5%
momentum is 0.7
rprop init. updates are 0.1
learning rate is 1.0
maximum epoch counter is 10000
networks are constructed with random weights [-1, 1).
a xor network has 2 input + 1 input bias, 3 hidden + 1 bias and 1 output neurons, sigmoid activation function everywhere
RMS error
-------------------------------
Online:    96% success rate (global error is under error tresshold)
           1100 iterations averaged (it is only accumulated on successful trainings, since failed runs have max iterations)

Batch:     99% success rate
           770 iterations average

Resilient: 80% success rate
           40 iterations average

The online learning seems to be the slowest. Resilient is pretty fast, but it often converges to 0.35 (maybe it is a local minima of this network, however I cannot find any proof about it). Batch is precise, and faster than online. Online even yields 0.011 error in 10k iterations, which is very slow.

Also, decresing the number of hidden units to 2 makes the convergence fail more often, but decreses the number of iterations on successful attempts.

Is there any free paper about the 3 hidden-layer XOR network (2 hidden-layer ffnn converges to this too), and if it has a local minima (at 0.35), or it is just my implementation problem? (I've found publications, but they are not free).

Answer 1

I tested XOR w/ RPROP, i=2, h=3, o=1, MSE, sigmoid, start weights [0:1].

Training data (while mse > 1%, epoch < 400) :

float trainingDataArr[] = {
    1.0f, 0.0f, 1.0f,
    0.0f, 0.0f, 0.0f,
    0.0f, 1.0f, 1.0f,
    1.0f, 1.0f, 0.0f
};

This network converged only ~83% of the time (out of 500 iterations). I'm not sure where your 0.35 is coming from (is that o average? o=0.35 regardless of i?, final rms?)

But, I can tell you that 80% convergence that you're experiencing is normal w/ 3 hidden nodes.

I'd recommend using a minimum of 4 (96% convergence) or even 5 (98% convergence) nodes in your hidden layer.

These results were generated using an RPROP algorithm as outlined in A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm by Professor Riedmiller[1].

Hope this helps and sorry I can't shed more light on the local minima.

http://deeplearning.cs.cmu.edu/pdfs/Rprop.pdf

UPDATE: Getting 93% convergence now using range [-1:1] for the random weight generation but I did have to normalize the inputs to get the std. deviation to 1 and mean to 0.0. Is it possible the 0.35 minima is occurring due to error translating between domain A with range 1 (0:1) and domain B with range 2 (-1:1) ?

void NeuralNet::NormalizeInputs(float* trainingData, int batchSize) {
  float max = FLT_MIN, min = FLT_MAX;

  for(int row = 0; row < batchSize*(nInput+nOutput); row+=(nInput+nOutput)) {
    for(int x = 0; x < nInput; x++) {
      if(trainingData[row+x] > max) max = trainingData[row+x];
      if(trainingData[row+x] < min) min = trainingData[row+x];
    }
  }

  for(int row = 0; row < batchSize*(nInput+nOutput); row+=(nInput+nOutput)) {
    for(int x = 0; x < nInput; x++) trainingData[row+x] = trainingData[row+x] * ( NEURAL_INPUT_RANGE/(max-min)  ) + 
      (NEURAL_INPUT_LOWER - (min*(NEURAL_INPUT_RANGE / (max-min))));
  }
}