I am training a neural network that needs to solve a regression problem.

To give some context, the network needs to predict a target value that lies in the range [0, 0.25] given as input features 4 time series. I have about 60000 examples, each consisting of 2500 time steps. The features are not standardized, but each of them has a small range (e.g. [-2, 2]).

The architecture I am testing is the following:

  • convolutional layer with 64 nodes, with activation = leaky relu
  • BatchNormalization
  • LSTM layer with 32 nodes
  • MaxPooling layer
  • Dense layer with 16 nodes, activation = relu
  • Dense layer with 1 node, activation = linear

The baseline errors one would make by creating a trivial model that always predicts the mean of the target variable are RMSE = 0.036 and MAE = 0.025 (where RMSE and MAE stand for RootMeanSquaredError and MeanAbsoluteError respectively).

No matter the optimization method and the learning rate I choose, the qualitative behavior of the loss function on the training set that I observe is always the same: in a few epochs the loss goes down to baseline level and then oscillates around it.

The loss on the test set either behaves similarly, but oscillations are more pronounced.

Can anybody guess why that happens? Is the problem too difficult to be learned with the current architecture?

I realize I have omitted many details and I am happy to provide them, if needed.

Thanks for the help!

  • $\begingroup$ Are all of the prediction values for the network "close to" the naive guess of the average value of the target? If so, this might be an instance of dying relu phenomenon. If it's not dying relu, and this behavior is repeated when using something like a leaky relu, then I'd experiment with different network designs -- this one doesn't seem to work, so you'll have to try something else. $\endgroup$ – Sycorax Jun 15 at 16:40
  • $\begingroup$ @Sycorax indeed the predictions lie very close to the naive guess. As a last attempt before giving up, following your suggestion, I am currently training the model with a leaky relu activation function on the last but one layer. Hope it helps! Thanks a lot, for now. $\endgroup$ – velvet Jun 16 at 7:11
  • $\begingroup$ Unfortunately switching to leaky ReLU activation function did not help: the training is qualitatively very similar to my previous attempts. I'll try to think whether a different architecture might help or if it's necessary to change model altogether. $\endgroup$ – velvet Jun 16 at 12:22
  • $\begingroup$ Wait why aren’t you scaling your data? $\endgroup$ – Sycorax Jun 16 at 12:55
  • $\begingroup$ Because their range of each feature vector is small (all values are in the range \approx [0, 2.5]) but, most of all, because standardizing them did not change the problem that I'm facing and I posted - that is, the network converges to a naive model :) Should I a always always always standardize features when solving regression problems? $\endgroup$ – velvet Jun 16 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.