making a surrogate model of a function using feed forward networks

Question

I have been trying to learn a very simple function using feedforward network(function and configuration of network below). I sample 10-12 points in the range [0,1] to train the network.

def fun1D(x):
return np.power(6*x-2,2)*np.sin(12*x-4)

model used: 2 hidden layers with 60 units, optmizer=Adam(learning_rate =0.01). code using keras as below:

def setupANN(dim=1,units=60,loss='mse'):
model = Sequential()

## making the model graph, Stacking layers is done by .add():
model.add(Dense(units=units, input_dim=dim, activation='sigmoid')) 
model.add(Dense(units=units, activation='sigmoid'))
model.add(Dense(units=1, activation= 'linear'))

optmiser = keras.optimizers.Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=0.0001, decay=0.0)

# configure the model's learning process; loss and optimisation etc
model.compile(loss=loss,
              optimizer=optmiser, metrics=["mae"])

return model

Training: epochs = 10000, batch_size = 5

After training, I predict values in the range [0, 1] and plot them. For most of the cases, I am getting a good representation of the original function. However, When I plot the training loss vs epoch, I get a curve like this, which is quite annoying. I even run a hyperparameter search to tune the network parameters (no. layers, units, optmiser, epochs) but most of them get me same result. I know this is counter-intuitive as the number of training points may be low for this. But I want to know if someone has any idea to improve it, especially the strange plot of loss vs epoch.

Answer 1

This is something that I have observed with the Adam Optimizer on training too much. Below is the training loss from one of my experiments.

After several such results, I was able to conclude that this was a result of training too much. My guess is that because Adam works by optimizing the learning rate parameter-wise, when the model is nearing a minima, and Adam requires to update a parameter that has been rarely updated, the effective learning rate applied to it is quite high. This leads to a high training loss which is followed by immediate recovery, leading to the spike. This is what you tend to see from thereon.

The common suggestions are:

1. Stop early (1500 epochs/steps in your case).
2. Switch to SGD when nearing the minima. This leads to a smoother approach to the minima at the end.