1
$\begingroup$

This question already has an answer here:

I am trying to train a simple neural network for regression, where the underlying function is a quadratic. Training data is generated by this underlying function, and I am just trying to get a network to learn this function.

This underlying function is $y = (x - 0.5)^2$, and I have generated 100 samples with $x$ from 0.0 to 1.0. Then, I train the network to take $x$ as input, and regress to the corresponding $y$. Training is done in batches, until the training loss converges.

However, my network cannot learn this function, even though it is very simple. These graphs show what is going on:

enter image description here

The top graph is the training data (one circle per datum), showing the underlying quadratic function. The second graph is the final network's predictions with the same inputs as the training data (i.e. 100 samples with $x$ from 0.0 to 1.0). The third graph overlays these two, and the fourth graph shows the training loss over time.

It is clear that the network is not learning the underlying function, and seems to be fitting a linear function to the best of its ability. However, my network has two hidden layers, with non-linear ReLU activations, and so should certainly be able to learn a quadratic function.

Here is the code for setting up and training the network:

# Create the neural  network
model = Sequential()
model.add(Dense(16, input_shape=(1,)))
model.add(Activation('relu'))
model.add(Dense(16, input_shape=(1,)))
model.add(Activation('relu'))
model.add(Dense(1))
sgd = SGD(lr=0.001)
model.compile(loss='mean_squared_error', optimizer=sgd)

# Create the training data
inputs = np.zeros((100, 1), dtype=np.float32)
targets = np.zeros((100, 1), dtype=np.float32)
for i in range(100):
    inputs[i] = 0.01 * i
    targets[i, 0] = np.power((inputs[i] - 0.5), 2)

# Training loop
while True:
    # Perform one weight update
    loss = model.train_on_batch(inputs, targets)
$\endgroup$

marked as duplicate by Reinstate Monica, Michael Chernick, kjetil b halvorsen, Peter Flom - Reinstate Monica Jul 5 '18 at 13:33

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ it would help to state which language it is written and which NN library was used $\endgroup$ – Aksakal Oct 11 '17 at 13:36
2
$\begingroup$

The problem is the activation function.

Note, the output is linear. So it shows the activation function (non-linear) part is not working well. Note, relu is $f(x)=\max (0,x)$, it is linear on $x>0$.

Instead of using relu using sigmoid or tanh may fix the problem.

$\endgroup$
  • $\begingroup$ In my experience sigmoid often performs better than tanh in TF. I know, it does not makes sense, but that's the fact. One can also try softmax, a smooth version of relu. $\endgroup$ – Ilya May 18 '17 at 19:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.