I understand the title (and the question itself) is a little generic, but I have some questions that I doubt I can find from google search or studying the topic (yes, I've tried for a while).
Basically, I was trying to approach neural networks with Python & keras with a very basic example. I'm trying to have a 2 layer neural network learn the xor operation. The script is the following:
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Dropout
x_train = np.random.randint(2, size=(1000,2))
y_train = np.array([[ 1 if x[0] != x[1] else 0] for x in x_train])
x_test = np.random.randint(2, size=(100,2))
y_test = np.array([[ 1 if x[0] != x[1] else 0] for x in x_test])
model = Sequential()
model.add(Dense(2, input_dim=2, activation="sigmoid"))
model.add(Dense(1, activation="sigmoid"))
model.compile(loss='binary_crossentropy',
optimizer='rmsprop',
metrics=['binary_accuracy'])
model.fit(x_train, y_train, batch_size=4,epochs=100, verbose=1)
score = model.evaluate(x_test, y_test, verbose=0)
print(score)
print(model.predict(np.array([[1,1],[1,0],[0,1],[0,0]])))
And it works as expected, achieving an accuracy of 1.0 around the 80th epoch.
From here I start to get confused, because I tried changing the activation function for the first layer to ReLU, but it doesn't work. The loss falls down to 0.3397 but then it stops progressing. Why is that? Isn't ReLU supposed to learn faster than Sigmoid (avoiding vanishing gradient)?
It would seems like a the network just isn't able to learn the xor function, like when I tried to use a single layer, which is mathematically unable to do it. However, I can't understand why it would be.
Moreover, this situation persists with 1 to 4 neurons in the hidden layer; if I add 5 of them, the network immediately jumps to perfect performance. Can someone explain the reason of this behaviour? It is probably something really obvious, but I can't seem to find it.