XOR with Neural Network [closed]

Question

I'm trying to implement a simple neural network to fit a XOR function as shown in the book 'Deep Learning' by Ian Goodfellow, Yoshua Bengio and Aaron Courville (2016). Here is my python code using Keras :

import keras
import numpy as np

# creating dataset
x = np.zeros((20000, 2), dtype=int)
x[5000:10000,1] += 1
x[10000:15000,0] += 1
x[15000:,:] += 1

np.random.shuffle(x)
y = [bit1 ^  bit2 for bit1, bit2 in x ]

x_train = x[:15000]
y_train = np.transpose(y[:15000])

x_test = x[15000:]
y_test = np.transpose(y[15000:])

# generating keras model
model = keras.models.Sequential([
  keras.layers.Dense(2, input_shape=(2,)),
  keras.layers.Dense(2, activation='relu'),
  keras.layers.Dense(1, activation='relu')
])

# compiling using stochastic gradient descent and MSE
model.compile(optimizer='SGD',
              loss='mean_squared_error')

# fit and evaluate
model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)

The issue is that I'm stuck with a loss equal to 0.5, does anyone know what I'm missing here ?

Answer 1

In addition to the other answer, note that you're currently training only on examples of x = [0,0], [0,1] and [1,0], and testing only on x = [1,1]. So your network actually has no way of learning the full XOR mapping, and wouldn't be expected to generalize to the test set. Also, you really don't need this many training examples, because you're just repeating identical information. Just all 4 possible inputs (combinations of 0 & 1) with their desired outputs would suffice. And then you can train using full batch gradient descent - no need to use SGD.

Answer 2

Firstly, you want to add a non-linearity to your first layer. If you don't mention it, you effectively only have one hidden layer. Secondly, because you expect your output to be either 0 or 1, you should treat this as a classification problem. So, modify the final activation to be 'sigmoid', and change your loss function to 'binary_crossentropy'. Finally, increase the number of epochs so that the network has enough time to learn.

Hope this helps!