I've trained many successful neural networks for classification problems. However, recently I'm working on some continuous input and output data, but no matter how I optimize my networks (structures, normalization, initialization, activation functions, optimization algorithms, regularization etc.), I always get inaccurate predictions with weird patterns. Here is an example:

Both input X and output Y are 100 dimension continuous data. The range of X is [-1,1]. I have 881 training data and 98 test data. Here is a preview of the data for the first 5 dimensions of X and Y:

Input X data distribution (first 5 dimensions) X data distribution (first 5 dimensions)

Output Y data distribution (first 5 dimensions Y data distribution (first 5 dimensions)

Here are the training results from one model. All predictions are severely shrinked, and show some weird patterns: it seems that any two variables are highly correlated and can be described by some nonlinear functions, but not randomly scattered as the ground truth.

Ground Truth Y for Training Ground Truth Y for Training

Prediction Y for training data Prediction Y

PS: This neural network has one hidden layer with 1000 hidden units, tanh as activation function, mean squared error loss, Xavier initialization, Adam optimization (lr = 0.002, beta_1 = 0.9, beta_2 = 0.999, schedule_decay = 0.004). (Yes, I also tried relu/sigmoid activation functions, more layers, different number of hidden units, regularization and dropouts, but these didn't change the results fundamentally.)

My questions are:

  1. Is this weird pattern (all predicted variables are correlated in some way) normal for all neural network training? And what causes this phenomenon?

  2. Is there anything further that I could try to improve the predictions (at least the range of prediction and ground truth are similar)? What causes this shrinkage?


migrated from stackoverflow.com Jul 18 '18 at 20:55

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ Did you scale your output? $\endgroup$ – Sycorax Aug 19 '18 at 23:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.