# making a surrogate model of a function using feed forward networks

I have been trying to learn a very simple function using feedforward network(function and configuration of network below). I sample 10-12 points in the range [0,1] to train the network.

def fun1D(x):
return np.power(6*x-2,2)*np.sin(12*x-4)


model used: 2 hidden layers with 60 units, optmizer=Adam(learning_rate =0.01). code using keras as below:

def setupANN(dim=1,units=60,loss='mse'):
model = Sequential()

## making the model graph, Stacking layers is done by .add():

optmiser = keras.optimizers.Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=0.0001, decay=0.0)

# configure the model's learning process; loss and optimisation etc
model.compile(loss=loss,
optimizer=optmiser, metrics=["mae"])

return model


Training: epochs = 10000, batch_size = 5

After training, I predict values in the range [0, 1] and plot them. For most of the cases, I am getting a good representation of the original function. However, When I plot the training loss vs epoch, I get a curve like this, which is quite annoying. I even run a hyperparameter search to tune the network parameters (no. layers, units, optmiser, epochs) but most of them get me same result. I know this is counter-intuitive as the number of training points may be low for this. But I want to know if someone has any idea to improve it, especially the strange plot of loss vs epoch.