0
$\begingroup$

I'm teaching a 3D convolutional neural network to learn different functions that map a 3D scalar field into another one. It is essentially a regression problem.

The distribution of input datasets does not change and essentially will get closer to a normal distribution. On the other hand, the outputs eventually may vary significantly in terms of distribution and range of possible min and max values. I'm training different networks to learn different output sets.

Which brings some questions: What would be the best approach for feature normalization in this case? Should I consider one type of normalization for inputs and a different one for outputs? Make sense to use any approach to fit the output scalar fields to a normal distribution? How do I achieve this?

Please consider the following histograms. Blue represents a common distribution for inputs and green two possible outputs, trained separately at different networks.

Min-max normalization: enter image description here

Standard-scale: enter image description here

When using min-max normalization, I'm getting values in a [0,1] range, but poorly distributed. Using a standard scale, min and max values are at unknown ranges but better distributed.

$\endgroup$
0
$\begingroup$

What would be the best approach for feature normalization in this case?

It depends on the range and shape of the data. You could consider non-static, trainable normalizations like batch-normalization and layer-normalization to prevent overfitting and to exploit faster convergence advantage of normalization techniques.

Should I consider one type of normalization for inputs and a different one for outputs?

Output normalization at training and de-normalization at predictions could lead to unintended errors, like change in evaluation accuracies on relative errors, or other non-linear accuracy metrics.

Make sense to use any approach to fit the output scalar fields to a normal distribution? How do I achieve this?

Not generally. With this, you'll constraint the network to indirectly fit at minimizing losses of values closer to mean / median. This could prevent the network from learning behaviors of rare segments of the population.

eg. learning a normalized income tax prediction might yield a model which fails to predict tax of super rich.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.