I have been trying to learn a very simple function using feedforward network(function and configuration of network below). I sample 10-12 points in the range [0,1] to train the network.
def fun1D(x): return np.power(6*x-2,2)*np.sin(12*x-4)
model used: 2 hidden layers with 60 units, optmizer=Adam(learning_rate =0.01). code using keras as below:
def setupANN(dim=1,units=60,loss='mse'): model = Sequential() ## making the model graph, Stacking layers is done by .add(): model.add(Dense(units=units, input_dim=dim, activation='sigmoid')) model.add(Dense(units=units, activation='sigmoid')) model.add(Dense(units=1, activation= 'linear')) optmiser = keras.optimizers.Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=0.0001, decay=0.0) # configure the model's learning process; loss and optimisation etc model.compile(loss=loss, optimizer=optmiser, metrics=["mae"]) return model
Training: epochs = 10000, batch_size = 5
After training, I predict values in the range [0, 1] and plot them. For most of the cases, I am getting a good representation of the original function. However, When I plot the training loss vs epoch, I get a curve like this, which is quite annoying. I even run a hyperparameter search to tune the network parameters (no. layers, units, optmiser, epochs) but most of them get me same result. I know this is counter-intuitive as the number of training points may be low for this. But I want to know if someone has any idea to improve it, especially the strange plot of loss vs epoch.