I've been interested in NNs for a while, just started playing with them. I liked the look of Keras, so I got started with some toycode to do some regression.

I tried the simplest set up I could:

  • $500$ inputs drawn uniformly from an interval $0<x<2*\pi$
  • Targets generated by $3\times sin(x)+1+e$ for $e\sim \mathcal{N}(0, 0.5)$ (Sin curve with random error).
  • Neural network therefore has one input ($x$) and one output ($y$).
  • One fully connected hidden layer with three neurons, activation function of $tanh$, loss function of mean squared error, training algorithm of stochastic gradient descent.

See self contained code gist.

The only problem is that it seems no matter what I try, I can't fit the values of $x$ for $pi<x<2*\pi$. I've tried different loss functions, Nesterov momentum, more epochs, more layers, more neurons, different learning rates, decay, momentum, different optimizers. Some made the fit slightly better, many changes made it worse.

Am I making a syntactical mistake somewhere that's affecting my results or is there a better way to build a simple MLP for this kind of problem? This is the first time I'm using Theano and Keras, so I don't know whether a fundamental mistake is plaguing me or if I need a new approach. I can't find any examples for regression using the Keras library.

  • $\begingroup$ One fully connected hidden layer with three neurons please try ~10 neurons. $\endgroup$
    – Alleo
    Commented Feb 2, 2016 at 13:20
  • $\begingroup$ @alleo I did, it didn't make a difference. $\endgroup$
    – Ogaday
    Commented Feb 2, 2016 at 13:39

2 Answers 2


If you want to model a sinusoid, I think that a stateful LSTM (RNN) might be a more natural choice. You can find an excellent example of modelling a sinusoid with an exponential amplitude decay in the keras example.

However, I tried out your Keras code, and I think your problem is that you're not letting it train long enough. Look at your loss at epoch 250, its VERY high!!

Epoch 250/250
360/360 [==============================] - 0s - loss: 0.5291 - val_loss: 0.7775

If I changed the number of nodes in your hidden layer to 10 and let it run for 15000 epochs instead of 250, I found that the loss was considerably lower and the plot more what you expect.

Epoch 15000/15000
360/360 [==============================] - 0s - loss: 0.2434 - val_loss: 0.2638

enter image description here

The updated code looks like:

# Multilayer Perceptron
model = Sequential()    # Feedforward
model.add(Dense(10, input_dim=1))
model.compile('sgd', 'mse')

hist = model.fit(xtr, ttr, validation_split=0.1, nb_epoch=15000)
  • $\begingroup$ You're absolutely correct, if I simply train for more epochs, the regression works very well! It seems I need to set the number of epochs in advance and there's no built in way to poll for convergence. I assume different parameter combinations will converge at different speeds? $\endgroup$
    – Ogaday
    Commented Feb 10, 2016 at 9:07
  • 1
    $\begingroup$ I'm glad that worked. Can you please mark your question as answered? cheers! $\endgroup$ Commented Feb 10, 2016 at 19:43
  • 2
    $\begingroup$ There are ways to "automatically" know when you're converged. Look up early stopping. Practically speaking, Keras' Sequential() model needs to know the number of epochs, but I've seen a couple of examples where you can have your model.fit() call be in a loop. After each epoch, you check your loss function's decrease, and if it is not low enough, keep going, once it starts to slow or hits an acceptable threshold, you can break out of the loop. Have a look at this: github.com/fchollet/hualos to help better see how the training is going! Good luck! $\endgroup$ Commented Feb 10, 2016 at 19:54
  • $\begingroup$ Great, I see the documentation now. Thanks for setting me on the right track. $\endgroup$
    – Ogaday
    Commented Feb 10, 2016 at 19:59

After line 44 add: model.add(Activation('linear'))

And initiate random weights as: def my_init(shape, name=None): value = np.random.random(shape) return K.variable(value, name=name)

so change line 42 as: model.add(Dense(3, input_dim=1, init=my_init))

  • 3
    $\begingroup$ This is rather cryptic, can you expand on why you suggest these changes and what their implications are? $\endgroup$
    – mdewey
    Commented Oct 21, 2016 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.