# Tensorflow batch normalization for images - padding issue

I'm trying to train anomaly/defect detection network on custom images. Let say I have to detect scratches on special steel boxes and I have two views:

• side view with dimension 2300 x 550 (width x height)
• top view with dimension 2300 x 1650 (width x height)

Both views are different (you see a bit different structure from side view and from top view). But it should not be a problem for state-of-the-art object detection networks. The issue I have to handle is different shape of images because I want to feed them into the same network.

I decided to set input shape to 1920 x 1080 (to fit it in GPU memory and lose as little resolution as possible). For input images I used this function:

new_image = tf.image.resize_image_with_pad(image, 1080, 1920)


This function resizes an input image to the target shape by keeping aspect ratio and the rest (to match the shape) is padded with zeros. The output of this function for the given two kinds of views results in:

• side view having padding on top and bottom
• top view having padding on left and right

Very simple example, not perfect just for imagination (side view on left, top view on right, black is padding):

Basic training setup:

• input shape: 1920 x 1080
• backbone network: resnet
• batch size: 1 (not able to make it bigger because of GPU memory limitation)

For training, everything went well, loss function, metrics, ... But for inference I got a bit strange results. For side views it worked quite well, for top views I got really strange results (output mask containing big parts everywhere). If I switch to is_training=True, output masks are much better.

I think the problem lays in batch normalization and its calculation of moving mean and variance (difference between batch and population statistics). The truth is that I have a bit more images of side views (like 60 % to 40 %) and if I train model only on top view, there is no issue. If I train it only on side views I get the same issue when doing inference on top views.

My question is - do you know how to solve this issue? I know I have option to train it separately. Or use different solution without padding. But I would be really interested how to train it together. I was thinking about "masking input solution" similar to this one. But how to propagate that to batch normalization layer in Tensorflow? How to mask it properly to have working solution? Do you have any experience with that? Or is there any strong reason it will never work well with padding?