4
$\begingroup$

I've implemented a neural network and I'm training it to compute Xor. 1 out of x times it fails to learn, where x is about 5 or 10. It then gives e.g. 0.67 instead of 0 as output for input (1,1). Is this just some unlucky randomization of the initial weights and should I move on to my real problem instance, or should I solve this first? What could be the cause?

Some more background info:

I'm using f(x) = 1/(1+exp(-x)) as activation function for both hidden neurons and output neuron. The hidden and output neuron have a bias. All weights are initially random numbers between 0 and 1. I'm using the backpropagation algorithm as described here: https://en.wikipedia.org/wiki/Backpropagation

I varied the learning factor from 0.001 to 1 and I did up to 1,000,000 training iterations.

$\endgroup$
1
  • $\begingroup$ I think it is possible if you are starting from very bad starting conditions. It also depends on the structure of the network and the data used to train it. $\endgroup$
    – Donbeo
    May 20 '16 at 17:41
3
$\begingroup$

Yes.

There are 16 local minimums that have the highest conversion if the weights are initialized between 0.5 and 1.

enter image description here

Image source: Yoshio Hirose, Koichi Yamashita, Shimpei Hijiya, "Back-propagation algorithm which varies the number of hidden units," Neural Networks, Volume 4, Issue 1, (1991)

$\endgroup$
0
0
$\begingroup$

I've got analog problem, when I was looking for the minimal neuron network architecture required to learn XOR which should be a (2,2,1) network. In fact, maths shows that the (2,2,1) network (2 entries, 2 neurons in the hidden layer, 1 output neuron) can solve the XOR problem, but maths doesn't show that the (2,2,1) network is easy to train. That said, I've got easily good results with (2,3,1) or (2,4,1) network architectures.

The problem seems to be related to the existence of many local minima. Look at this 1998 paper, «Learning XOR: exploring the space of a classic problem» by Richard Bland. Maybe you could try different random initialization of the weights or change your loss function.

It works fine with Keras or TensorFlow using loss function 'mean_squared_error', sigmoid activation and Adam optimizer. Furthermore weights initialization with random number between 0.5 and 1.0 helps to converge. Even with pretty good hyperparameters, I observed that the learned XOR model is trapped in a local minimum about 15% of the time.

$\endgroup$
1
  • $\begingroup$ This is very similar to the answer here & here. Please do not post duplicate answers. If you believe a question is completely answered by an answer elsewhere, flag / vote to close that question as a duplicate of the other. If it isn't completely answered by the other answer, customize your answer here to be more specific to the question. $\endgroup$ Sep 13 '20 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.