How to perform feature normalization for training regression CNNs with datasets with different distributions?

Question

I'm teaching a 3D convolutional neural network to learn different functions that map a 3D scalar field into another one. It is essentially a regression problem.

The distribution of input datasets does not change and essentially will get closer to a normal distribution. On the other hand, the outputs eventually may vary significantly in terms of distribution and range of possible min and max values. I'm training different networks to learn different output sets.

Which brings some questions: What would be the best approach for feature normalization in this case? Should I consider one type of normalization for inputs and a different one for outputs? Make sense to use any approach to fit the output scalar fields to a normal distribution? How do I achieve this?

Please consider the following histograms. Blue represents a common distribution for inputs and green two possible outputs, trained separately at different networks.

Min-max normalization:

Standard-scale:

When using min-max normalization, I'm getting values in a [0,1] range, but poorly distributed. Using a standard scale, min and max values are at unknown ranges but better distributed.

Answer 1

What would be the best approach for feature normalization in this case?

It depends on the range and shape of the data. You could consider non-static, trainable normalizations like batch-normalization and layer-normalization to prevent overfitting and to exploit faster convergence advantage of normalization techniques.

Should I consider one type of normalization for inputs and a different one for outputs?

Output normalization at training and de-normalization at predictions could lead to unintended errors, like change in evaluation accuracies on relative errors, or other non-linear accuracy metrics.

Make sense to use any approach to fit the output scalar fields to a normal distribution? How do I achieve this?

Not generally. With this, you'll constraint the network to indirectly fit at minimizing losses of values closer to mean / median. This could prevent the network from learning behaviors of rare segments of the population.

eg. learning a normalized income tax prediction might yield a model which fails to predict tax of super rich.