I have a training set of 6400 samples. Each sample is composed of an input of size 100, which is essentially a noise process. The input of the first sample is:

enter image description here

The output is the solution of a differential equation for the noise (with average of zero over all samples). The corresponding output for the first sample is:

enter image description here

For the comparison, I tried to train a feed forward neural network in tensorflow. The minimal code is:

import tensorflow            as tf
from sklearn.model_selection import train_test_split

#normalize data
inp_data = tf.keras.utils.normalize(inp_data)
out_data = tf.keras.utils.normalize(out_data)

#split data into train, val and test sets
inp_train, inp_test, out_train, out_test = train_test_split(inp_data, out_data, test_size=0.33, random_state= 1234)
inp_val, inp_test, out_val, out_test     = train_test_split(inp_test, out_test, test_size=0.5, random_state= 1234)

#build a feed-forward NN
act = 'relu'
ini = tf.keras.initializers.GlorotUniform(seed=1234)
reg = None
opt = tf.keras.optimizers.Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, amsgrad=False)
model = tf.keras.models.Sequential()
#hidden layer
model.add(tf.keras.layers.Dense(units=1, activation=act, kernel_initializer=ini, kernel_regularizer=reg, input_shape=(inp_train.shape[1], )))
#output layer
model.add(tf.keras.layers.Dense(units=out_train.shape[1], activation='linear', kernel_initializer=ini, kernel_regularizer=reg))
model.compile(optimizer=opt, loss='mae')
model.fit(inp_train, out_train, shuffle=True, epochs=100, batch_size=10, validation_data=(inp_val, out_val), verbose=1)

The network contains 301 parameters. The training proceed as:

Train on 4288 samples, validate on 1056 samples
Epoch 1/100
4288/4288 [==============================] - 2s 550us/sample - loss: 0.0340 - val_loss: 0.0283
Epoch 2/100
4288/4288 [==============================] - 2s 440us/sample - loss: 0.0285 - val_loss: 0.0282
Epoch 3/100
4288/4288 [==============================] - 2s 369us/sample - loss: 0.0284 - val_loss: 0.0281
Epoch 4/100
4288/4288 [==============================] - 2s 431us/sample - loss: 0.0283 - val_loss: 0.0281
Epoch 5/100
4288/4288 [==============================] - 2s 418us/sample - loss: 0.0283 - val_loss: 0.0281
Epoch 6/100
4288/4288 [==============================] - 2s 391us/sample - loss: 0.0283 - val_loss: 0.0281

The resulting learning curve is:

enter image description here

So, with just one neuron in hidden layer, the network converges very fast. But the prediction on the test set is off.

What I tried so far:

1 - change activation function (to tanh)

2 - change initializer (to normal distribution)

3 - increase number of neurons in hidden layer (to 10)

4 - add a regulizer (L1/L2)

5 - change optimizer (to SGD)

They gave the same result. My guess is that it got stuck in a local minima, but I am not sure. How should I interpret the learning curve and any suggestions on how to improve the network?


Your normalization process doesn't look correct to me, you first need to normalize the training data, then use these normalization parameters to scale the validation and test sets. Also, are you sure you need to normalize the outputs? I'm not saying that these are the cause of your problem but maybe you can give it a try.

Judging from the trend of your training loss it doesn't look like your network is actually learning something since the difference between the initial loss with random parameters and the final loss with your final parameters is small (and the latter is basically the loss after one epoch). I'd try to play a bit with the hyperparameters, e.g. increasing first of all the learning rate, then the number of hidden neurons (push a bit further!) and hidden layers.

  • $\begingroup$ I tried without normalizing the data. It gives the same trend but at a higher value. I also increased the learning rate. It did not change much, until learning _rate=1, in which the learning curves shows oscillatory behavior. I also increased the number of hidden neurons (to 100), and independently the number of hidden layers (to 10). Does not help much. $\endgroup$ – New Developer Feb 27 '20 at 10:04
  • $\begingroup$ Ok, then I'd suggest to try with the correct normalization procedure. If this also doesn't work check the effect of increasing the batch size and the number of hidden neurons up to a 1000. $\endgroup$ – black_cat Feb 27 '20 at 10:12
  • 1
    $\begingroup$ With the correct normalization procedure (explained in datascience.stackexchange.com/questions/27615/…), it worked. Now, I see a decent decrease of the error in the learning curves. I'll proceed from there. $\endgroup$ – New Developer Feb 27 '20 at 10:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.