I made a 'mistake' while training a neural network, it is a typical image classification problem like this. However the data is much larger and came from Kaggle.

In my Dataset class from PyTorch, I defined a flag

if self.transform_norm is False:
    image = image.astype(np.float32) / 255.0

and this would signify that if my augmentations pipeline does not have a normalization technique, then we set this flag to False. One example that I would set the above flag to be True the below augmentation appear in my pipeline:

albumentations.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0)

I forgot to set the flag to True and thus, the images first went through a standardization from [0,255] to [0,1] and then normalized using mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]. I thought I did wrong but the training results were actually good. So I dug deeper and found PyTorch's documentation and realized that it may be me who has been doing it all wrong? However, I am not using PyTorch's pretrained model out of the box, usually, I go to Ross's timm/geffnet for the models. Do let me know if there is a "right" approach.

To quote the link:

All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. You can use the following transform to normalize:

  • $\begingroup$ The quote seems to describe what you’ve done. What part of it makes you uncertain? $\endgroup$ – Sycorax Dec 28 '20 at 14:20
  • $\begingroup$ Thanks! The uncertainty comes from the part whereby for my past trainings (many of them) I have almost never intentionally do both /255 and normalization as described above at the same time. I always thought it is either or. But I realised I may be wrong. $\endgroup$ – nan Dec 28 '20 at 14:50
  • 1
    $\begingroup$ I agree that this is unusual, but it seems that, in this particular case, the documentation is very explicit about scaling in this exact way. As a matter of mathematics, linear functions are closed under composition, so if you really wanted to, you could roll both steps into a single step using a little arithmetic. $\endgroup$ – Sycorax Dec 28 '20 at 15:45
  • $\begingroup$ I agree, thanks for your reply. I was actually shocked that the end results are good. (I am not using the pretrained models from the documentation so I don’t think I need to obey this). But I’m just confused on why it worked. $\endgroup$ – nan Dec 28 '20 at 15:52
  • 1
    $\begingroup$ So perhaps you could edit your question to ask "Why does this work?" instead of asking whether it's right, since you have documentation that very plainly states that this is correct. $\endgroup$ – Sycorax Dec 28 '20 at 16:06

It's not normalized twice! It's normalized once, using two steps.

I say it's normalized once because all the normalization does is apply two linear transformations. We can rewrite two or more linear transformations as a single linear transformation. All we need to do is to choose a certain linear function to transform the raw pixel values to the values that the neural network expects to receive. We can show that the two-step process achieved in code is exactly the same as an equivalent operation carried out in one step.

The network expects to receive images with a certain scale, but images are encoded by with values between 0 and 255. The route suggested in the documentation is two-step.

  1. For some pixel $p\in[0,255]$, divide by 255: $q = \frac{p}{255}$. If we wanted to emphasize that this is a "linear scale and shift" transformation, we could even write $q = \frac{p-0}{255}$.
  2. Subtract the mean and divide by the standard deviation: $z = \frac{q - \mu}{\sigma}$. The network expects to receive $z$ as inputs.

We can do this in one step instead, because the composition of linear functions is linear. Just doing substitution and rearranging, we can show

$$\begin{align} z&=\frac{q - \mu}{\sigma} \\ &=\frac{p/255 - \mu}{\sigma} \\ &=\frac{p - 255\mu}{255\sigma} \\ \end{align} $$

So in the specific case, you can achieve the exact same scaling using the transformation albumentations.Normalize(mean=[255*0.485, 255*0.456, 255*0.406], std=[255*0.229, 255*0.224, 255*0.225]). This should make intuitive sense, because we're just rescaling the transformation to take place on the $[0,255]$ interval of pixel values, instead of a $[0,1]$ interval.

  • $\begingroup$ I understood perfectly now thanks! But just one more question, why did my accidental normalization "thrice" worked, I went through one division of 255, followed another Normalize() which is another division of 255, and then applying the mean and std. So technically, if my image pixel is 255, I went through 3 transformations, albeit all linear. $\endgroup$ – nan Dec 29 '20 at 4:37
  • 1
    $\begingroup$ If you divided by 255 twice, then it's not consistent with the description provided in the documentation. Therefore, I wonder if it did, actually, work -- my assumption is that following the documentation would provide a superior result. But I don't know what the class albumentations.Normalize does; it's not a part of pytorch. Since your question has changed (divide 1 vs 2 times), but this already has an answer, you could ask a new question. $\endgroup$ – Sycorax Dec 29 '20 at 4:51
  • $\begingroup$ Silly question: Normalizing a collection of values to a given mean and std gives the exact same result regardless of prior scaling. So scaling to [0,1] should have no effect mathematically. Is it maybe all a question of data types (float vs int) or something? $\endgroup$ – FaultyBagnose May 5 at 6:29
  • $\begingroup$ @FaultyBagnose I don’t understand what you’re asking. Scaling to a unit interval is different than scaling to have 0 mean and unit variance. If you have a new question, you can ask it using the ASK QUESTION button at the top of the page. $\endgroup$ – Sycorax May 5 at 6:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.