Clarification on simple perceptron neural network

Question

I understand the title (and the question itself) is a little generic, but I have some questions that I doubt I can find from google search or studying the topic (yes, I've tried for a while).

Basically, I was trying to approach neural networks with Python & keras with a very basic example. I'm trying to have a 2 layer neural network learn the xor operation. The script is the following:

import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Dropout

x_train = np.random.randint(2, size=(1000,2))
y_train = np.array([[ 1 if x[0] != x[1] else 0] for x in x_train])

x_test = np.random.randint(2, size=(100,2))
y_test = np.array([[ 1 if x[0] != x[1] else 0] for x in x_test])

model = Sequential()
model.add(Dense(2, input_dim=2, activation="sigmoid"))
model.add(Dense(1, activation="sigmoid"))

model.compile(loss='binary_crossentropy',
          optimizer='rmsprop',
          metrics=['binary_accuracy'])

model.fit(x_train, y_train, batch_size=4,epochs=100, verbose=1)

score = model.evaluate(x_test, y_test, verbose=0)
print(score)

print(model.predict(np.array([[1,1],[1,0],[0,1],[0,0]])))

And it works as expected, achieving an accuracy of 1.0 around the 80th epoch.

From here I start to get confused, because I tried changing the activation function for the first layer to ReLU, but it doesn't work. The loss falls down to 0.3397 but then it stops progressing. Why is that? Isn't ReLU supposed to learn faster than Sigmoid (avoiding vanishing gradient)?

It would seems like a the network just isn't able to learn the xor function, like when I tried to use a single layer, which is mathematically unable to do it. However, I can't understand why it would be.

Moreover, this situation persists with 1 to 4 neurons in the hidden layer; if I add 5 of them, the network immediately jumps to perfect performance. Can someone explain the reason of this behaviour? It is probably something really obvious, but I can't seem to find it.

Answer 1

The important thing to keep in mind about the mathematical proof pertaining to neural networks and the XOR problem is that the proof merely shows that a network exists that can solve the XOR problem, but it does not show that such a network is easy to learn.

Tensor playground gives a nice interface into several toy neural networks problems, and an XOR-like problem is one of them. Using 1 hidden layer with 2 neurons shows that this is a pretty terrible approach with relus. But using a 4-2 relu network works really well!

Try it out!