I have been trying to learn a very simple function using feedforward network(function and configuration of network below). I sample 10-12 points in the range [0,1] to train the network.

def fun1D(x):
return np.power(6*x-2,2)*np.sin(12*x-4)

model used: 2 hidden layers with 60 units, optmizer=Adam(learning_rate =0.01). code using keras as below:

def setupANN(dim=1,units=60,loss='mse'):
model = Sequential()

## making the model graph, Stacking layers is done by .add():
model.add(Dense(units=units, input_dim=dim, activation='sigmoid')) 
model.add(Dense(units=units, activation='sigmoid'))
model.add(Dense(units=1, activation= 'linear'))

optmiser = keras.optimizers.Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=0.0001, decay=0.0)

# configure the model's learning process; loss and optimisation etc
              optimizer=optmiser, metrics=["mae"])

return model

Training: epochs = 10000, batch_size = 5

After training, I predict values in the range [0, 1] and plot them. For most of the cases, I am getting a good representation of the original function. However, When I plot the training loss vs epoch, I get a curve like this, which is quite annoying. I even run a hyperparameter search to tune the network parameters (no. layers, units, optmiser, epochs) but most of them get me same result. I know this is counter-intuitive as the number of training points may be low for this. But I want to know if someone has any idea to improve it, especially the strange plot of loss vs epoch. enter image description here


1 Answer 1


This is something that I have observed with the Adam Optimizer on training too much. Below is the training loss from one of my experiments. enter image description here

After several such results, I was able to conclude that this was a result of training too much. My guess is that because Adam works by optimizing the learning rate parameter-wise, when the model is nearing a minima, and Adam requires to update a parameter that has been rarely updated, the effective learning rate applied to it is quite high. This leads to a high training loss which is followed by immediate recovery, leading to the spike. This is what you tend to see from thereon.

The common suggestions are:

  1. Stop early (1500 epochs/steps in your case).
  2. Switch to SGD when nearing the minima. This leads to a smoother approach to the minima at the end.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.