I've trained many successful neural networks for classification problems. However, recently I'm working on some continuous input and output data, but no matter how I optimize my networks (structures, normalization, initialization, activation functions, optimization algorithms, regularization etc.), I always get inaccurate predictions with weird patterns. Here is an example:
Both input X and output Y are 100 dimension continuous data. The range of X is [-1,1]. I have 881 training data and 98 test data. Here is a preview of the data for the first 5 dimensions of X and Y:
Here are the training results from one model. All predictions are severely shrinked, and show some weird patterns: it seems that any two variables are highly correlated and can be described by some nonlinear functions, but not randomly scattered as the ground truth.
PS: This neural network has one hidden layer with 1000 hidden units, tanh as activation function, mean squared error loss, Xavier initialization, Adam optimization (lr = 0.002, beta_1 = 0.9, beta_2 = 0.999, schedule_decay = 0.004). (Yes, I also tried relu/sigmoid activation functions, more layers, different number of hidden units, regularization and dropouts, but these didn't change the results fundamentally.)
My questions are:
Is this weird pattern (all predicted variables are correlated in some way) normal for all neural network training? And what causes this phenomenon?
Is there anything further that I could try to improve the predictions (at least the range of prediction and ground truth are similar)? What causes this shrinkage?