Is it normal that a Neural Network sometimes doesn't learn Xor? I've implemented a neural network and I'm training it to compute Xor. 1 out of x times it fails to learn, where x is about 5 or 10. It then gives e.g. 0.67 instead of 0 as output for input (1,1). Is this just some unlucky randomization of the initial weights and should I move on to my real problem instance, or should I solve this first? What could be the cause?
Some more background info:
I'm using f(x) = 1/(1+exp(-x)) as activation function for both hidden neurons and output neuron. The hidden and output neuron have a bias. All weights are initially random numbers between 0 and 1. I'm using the backpropagation algorithm as described here: https://en.wikipedia.org/wiki/Backpropagation
I varied the learning factor from 0.001 to 1 and I did up to 1,000,000 training iterations.
 A: Yes.
There are 16 local minimums that have the highest conversion if the weights are initialized between 0.5 and 1.

Image source: Yoshio Hirose, Koichi Yamashita, Shimpei Hijiya, "Back-propagation algorithm which varies the number of hidden units," Neural Networks, Volume 4, Issue 1, (1991)

*

*A similar question including an implementation in tflearn
A: I've got analog problem, when I was looking for the minimal neuron network architecture required to learn XOR which should be a (2,2,1) network. In fact, maths shows that the (2,2,1) network (2 entries, 2 neurons in the hidden layer, 1 output neuron) can solve the XOR problem, but maths doesn't show that the (2,2,1) network is easy to train. That said, I've got easily good results with (2,3,1) or (2,4,1) network architectures.
The problem seems to be related to the existence of many local minima. Look at this 1998 paper, «Learning XOR: exploring the space of a classic problem» by Richard Bland. Maybe you could try different random initialization of the weights or change your loss function.
It works fine with Keras or TensorFlow using loss function 'mean_squared_error', sigmoid activation and Adam optimizer. Furthermore weights initialization with random number between 0.5 and 1.0 helps to converge. Even with pretty good hyperparameters, I observed that the learned XOR model is trapped in a local minimum about 15% of the time.
