I am training a neural network that needs to solve a regression problem.
To give some context, the network needs to predict a target value that lies in the range [0, 0.25] given as input features 4 time series. I have about 60000 examples, each consisting of 2500 time steps. The features are not standardized, but each of them has a small range (e.g. [-2, 2]).
The architecture I am testing is the following:
- convolutional layer with 64 nodes, with activation = leaky relu
- LSTM layer with 32 nodes
- MaxPooling layer
- Dense layer with 16 nodes, activation = relu
- Dense layer with 1 node, activation = linear
The baseline errors one would make by creating a trivial model that always predicts the mean of the target variable are RMSE = 0.036 and MAE = 0.025 (where RMSE and MAE stand for RootMeanSquaredError and MeanAbsoluteError respectively).
No matter the optimization method and the learning rate I choose, the qualitative behavior of the loss function on the training set that I observe is always the same: in a few epochs the loss goes down to baseline level and then oscillates around it.
The loss on the test set either behaves similarly, but oscillations are more pronounced.
Can anybody guess why that happens? Is the problem too difficult to be learned with the current architecture?
I realize I have omitted many details and I am happy to provide them, if needed.
Thanks for the help!