This question already has an answer here:
I am trying to train a simple neural network for regression, where the underlying function is a quadratic. Training data is generated by this underlying function, and I am just trying to get a network to learn this function.
This underlying function is $y = (x - 0.5)^2$, and I have generated 100 samples with $x$ from 0.0 to 1.0. Then, I train the network to take $x$ as input, and regress to the corresponding $y$. Training is done in batches, until the training loss converges.
However, my network cannot learn this function, even though it is very simple. These graphs show what is going on:
The top graph is the training data (one circle per datum), showing the underlying quadratic function. The second graph is the final network's predictions with the same inputs as the training data (i.e. 100 samples with $x$ from 0.0 to 1.0). The third graph overlays these two, and the fourth graph shows the training loss over time.
It is clear that the network is not learning the underlying function, and seems to be fitting a linear function to the best of its ability. However, my network has two hidden layers, with non-linear ReLU activations, and so should certainly be able to learn a quadratic function.
Here is the code for setting up and training the network:
# Create the neural network model = Sequential() model.add(Dense(16, input_shape=(1,))) model.add(Activation('relu')) model.add(Dense(16, input_shape=(1,))) model.add(Activation('relu')) model.add(Dense(1)) sgd = SGD(lr=0.001) model.compile(loss='mean_squared_error', optimizer=sgd) # Create the training data inputs = np.zeros((100, 1), dtype=np.float32) targets = np.zeros((100, 1), dtype=np.float32) for i in range(100): inputs[i] = 0.01 * i targets[i, 0] = np.power((inputs[i] - 0.5), 2) # Training loop while True: # Perform one weight update loss = model.train_on_batch(inputs, targets)