Regression for noisy data with tensorflow: low train and validation errors but high test error

I have a training set of 6400 samples. Each sample is composed of an input of size 100, which is essentially a noise process. The input of the first sample is:

The output is the solution of a differential equation for the noise (with average of zero over all samples). The corresponding output for the first sample is:

For the comparison, I tried to train a feed forward neural network in tensorflow. The minimal code is:

import tensorflow            as tf
from sklearn.model_selection import train_test_split

#normalize data
inp_data = tf.keras.utils.normalize(inp_data)
out_data = tf.keras.utils.normalize(out_data)

#split data into train, val and test sets
inp_train, inp_test, out_train, out_test = train_test_split(inp_data, out_data, test_size=0.33, random_state= 1234)
inp_val, inp_test, out_val, out_test     = train_test_split(inp_test, out_test, test_size=0.5, random_state= 1234)

#build a feed-forward NN
#hyperparameters
act = 'relu'
ini = tf.keras.initializers.GlorotUniform(seed=1234)
reg = None
#model
model = tf.keras.models.Sequential()
#hidden layer
model.add(tf.keras.layers.Dense(units=1, activation=act, kernel_initializer=ini, kernel_regularizer=reg, input_shape=(inp_train.shape[1], )))
#output layer
model.compile(optimizer=opt, loss='mae')
model.fit(inp_train, out_train, shuffle=True, epochs=100, batch_size=10, validation_data=(inp_val, out_val), verbose=1)


The network contains 301 parameters. The training proceed as:

Train on 4288 samples, validate on 1056 samples
Epoch 1/100
4288/4288 [==============================] - 2s 550us/sample - loss: 0.0340 - val_loss: 0.0283
Epoch 2/100
4288/4288 [==============================] - 2s 440us/sample - loss: 0.0285 - val_loss: 0.0282
Epoch 3/100
4288/4288 [==============================] - 2s 369us/sample - loss: 0.0284 - val_loss: 0.0281
Epoch 4/100
4288/4288 [==============================] - 2s 431us/sample - loss: 0.0283 - val_loss: 0.0281
Epoch 5/100
4288/4288 [==============================] - 2s 418us/sample - loss: 0.0283 - val_loss: 0.0281
Epoch 6/100
4288/4288 [==============================] - 2s 391us/sample - loss: 0.0283 - val_loss: 0.0281


The resulting learning curve is:

So, with just one neuron in hidden layer, the network converges very fast. But the prediction on the test set is off.

What I tried so far:

1 - change activation function (to tanh)

2 - change initializer (to normal distribution)

3 - increase number of neurons in hidden layer (to 10)

4 - add a regulizer (L1/L2)

5 - change optimizer (to SGD)

They gave the same result. My guess is that it got stuck in a local minima, but I am not sure. How should I interpret the learning curve and any suggestions on how to improve the network?