Neural Network cannot learn a simple function [duplicate]

I am trying to train a simple neural network for regression, where the underlying function is a quadratic. Training data is generated by this underlying function, and I am just trying to get a network to learn this function.

This underlying function is $y = (x - 0.5)^2$, and I have generated 100 samples with $x$ from 0.0 to 1.0. Then, I train the network to take $x$ as input, and regress to the corresponding $y$. Training is done in batches, until the training loss converges.

However, my network cannot learn this function, even though it is very simple. These graphs show what is going on:

The top graph is the training data (one circle per datum), showing the underlying quadratic function. The second graph is the final network's predictions with the same inputs as the training data (i.e. 100 samples with $x$ from 0.0 to 1.0). The third graph overlays these two, and the fourth graph shows the training loss over time.

It is clear that the network is not learning the underlying function, and seems to be fitting a linear function to the best of its ability. However, my network has two hidden layers, with non-linear ReLU activations, and so should certainly be able to learn a quadratic function.

Here is the code for setting up and training the network:

# Create the neural  network
model = Sequential()
sgd = SGD(lr=0.001)
model.compile(loss='mean_squared_error', optimizer=sgd)

# Create the training data
inputs = np.zeros((100, 1), dtype=np.float32)
targets = np.zeros((100, 1), dtype=np.float32)
for i in range(100):
inputs[i] = 0.01 * i
targets[i, 0] = np.power((inputs[i] - 0.5), 2)

# Training loop
while True:
# Perform one weight update
loss = model.train_on_batch(inputs, targets)


marked as duplicate by Sycorax, Michael Chernick, kjetil b halvorsen, Peter Flom♦Jul 5 '18 at 13:33

Note, the output is linear. So it shows the activation function (non-linear) part is not working well. Note, relu is $f(x)=\max (0,x)$, it is linear on $x>0$.
Instead of using relu using sigmoid or tanh may fix the problem.