2
$\begingroup$

I am using 2 BatchNormalization layers in Keras for a huge dataset that does not fit into memory. I can train with normalized values, but since I want to do a regression, I need to reverse or undo the effects of the normalization, that is, after I do

prediction = model.predict(my_test)

I obtain values that are affected by the 2 BatchNormalizations, and I want to apply to prediction a function to show the values in their natural scale. How can I do this in Keras? Note that I have 2 layers of BatchNormalization.

(I know how to do it when the dataset fits into memory and I normalize it before the training without using BatchNormalization, and then de-normalize it, but in this case I am confused). Thx.

Addendum: By @Romain's suggestion: full model. Inspired by https://medium.com/@rajin250/precipitation-prediction-using-convlstm-deep-neural-network-b9e9b617b436 I want to predict rainfall based in previous rainfall images for a 150x150 points of a grid area, so I have data from 0 to unbounded values,

model = Sequential()
model.add(ConvLSTM2D(filters=32, kernel_size=(5, 5),
                     input_shape=(None, 150, 150, 1), padding='same',
                     return_sequences=True,
                     activation="tanh", recurrent_activation='hard_sigmoid',
                     kernel_initializer='glorot_uniform',
                     unit_forget_bias=True,
                     dropout=0.2, recurrent_dropout=0.2))
model.add(BatchNormalization())

model.add(ConvLSTM2D(filters=32, kernel_size=(2, 2), padding='same',
                     return_sequences=False,
                     activation="tanh", recurrent_activation='hard_sigmoid',
                     kernel_initializer='glorot_uniform',
                     unit_forget_bias=True,
                     dropout=0.2, recurrent_dropout=0.2))
model.add(BatchNormalization())

model.add(Conv2D(filters=1, kernel_size=(1, 1),
                 #activation="sigmoid",
                 padding='same', data_format='channels_last'))

model.compile(optimizer="adam", loss="mse")

I have made explicit that the activation is commented since the sigmoid forces my values to be between 0 and 1. The training is working but the values I obtain are very small, far from reality, so my thought was that I needed to de-normalize somehow...

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ I think you are having a mis conception about "Batch Normalization". This does not concern your input data but rather the output of each layer on a given batch of data. You don't need to "reverse" it. Can you give more detail about your architecture as well as your loss function please ? $\endgroup$
    – Romain
    Commented Mar 30, 2021 at 13:38
  • 2
    $\begingroup$ If predicted values are not close to the observed values, this implies that the training loss is large. In other words, the model isn’t fitting the data very well. $\endgroup$
    – Sycorax ♦
    Commented Mar 30, 2021 at 13:55
  • $\begingroup$ But the model is not far from predicting well. Using a sigmoid activation in the last layer, I can see the rainfall in more or less correct places, although I don't have exact values since the sigmoid is limiting them, but it encourages me to think that the model is seeing something. Curiously, without a sigmoid, (as it should be with "mse" as loss and a regression) the training worsens. $\endgroup$
    – David
    Commented Mar 30, 2021 at 14:00
  • 2
    $\begingroup$ I just saw the edit to your question. You should put a linear layer as the last part of the network. $\endgroup$
    – Sycorax ♦
    Commented Mar 30, 2021 at 14:06

1 Answer 1

Reset to default
1
$\begingroup$

If input data is not normalized (or standardized), it will not work in a Neural Network.

This is actually a numerical problem, due to the gradient update but in general one has to normalize or standardize data when dealing with deep learning.

So probably that it will work if you normalize your data and make the last layer a linear one rather than sigmoid.

Even if your data does not fit in memory, you can find ways to normalize.

If your input are images then you simply divide every pixel by 255.0. In keras you can do this using generator, you can find an exemple here: https://medium.com/@mrgarg.rajat/training-on-large-datasets-that-dont-fit-in-memory-in-keras-60a974785d71

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. I am working with the precipitation, a variable that is strongly non-gaussian, skewed and fat tailed. (It has been sometimes modeled as a Gamma distribution). I have checked that the outputs after an activation "relu" are all 0, and after an activation "linear" are all negative. This is probably due to the batch normalization layers, that bring all the values to negative values, and indeed it could explain why the author of the blog post used a sigmoid: because it converts to positive the negative values. $\endgroup$
    – David
    Commented Mar 31, 2021 at 17:59
  • $\begingroup$ I have looked at this to "standardize" skewed data (but not tried yet): datascience.stackexchange.com/questions/36291/…. Some of those solutions could solve the previous standardization of the dataset (before entering the NN), but I'm not sure if the internal batch normalization layers would spoil everything again. Perhaps some layer customization could be ideal, I should check if that can be done in Keras. $\endgroup$
    – David
    Commented Mar 31, 2021 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.