4
$\begingroup$

I'm trying to train anomaly/defect detection network on custom images. Let say I have to detect scratches on special steel boxes and I have two views:

  • side view with dimension 2300 x 550 (width x height)
  • top view with dimension 2300 x 1650 (width x height)

Both views are different (you see a bit different structure from side view and from top view). But it should not be a problem for state-of-the-art object detection networks. The issue I have to handle is different shape of images because I want to feed them into the same network.

I decided to set input shape to 1920 x 1080 (to fit it in GPU memory and lose as little resolution as possible). For input images I used this function:

new_image = tf.image.resize_image_with_pad(image, 1080, 1920)

This function resizes an input image to the target shape by keeping aspect ratio and the rest (to match the shape) is padded with zeros. The output of this function for the given two kinds of views results in:

  • side view having padding on top and bottom
  • top view having padding on left and right

Very simple example, not perfect just for imagination (side view on left, top view on right, black is padding):

side view on left, top view on right, black is padding

Basic training setup:

  • input shape: 1920 x 1080
  • GT - masks
  • backbone network: resnet
  • batch size: 1 (not able to make it bigger because of GPU memory limitation)

For training, everything went well, loss function, metrics, ... But for inference I got a bit strange results. For side views it worked quite well, for top views I got really strange results (output mask containing big parts everywhere). If I switch to is_training=True, output masks are much better.

I think the problem lays in batch normalization and its calculation of moving mean and variance (difference between batch and population statistics). The truth is that I have a bit more images of side views (like 60 % to 40 %) and if I train model only on top view, there is no issue. If I train it only on side views I get the same issue when doing inference on top views.

My question is - do you know how to solve this issue? I know I have option to train it separately. Or use different solution without padding. But I would be really interested how to train it together. I was thinking about "masking input solution" similar to this one. But how to propagate that to batch normalization layer in Tensorflow? How to mask it properly to have working solution? Do you have any experience with that? Or is there any strong reason it will never work well with padding?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

For similar reasons to what you've shown, batch norm doesn't work well with small batch sizes (less than 8 or so). Group normalization or instance normalization is the preferred substitute in that case.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I didn't hear much about GN or IN, do you have personal experience with it? I found quite nice comparison here. I agree batch norm doesn't work well with small batch sizes, but I would not expect this kind of error. I would expect something like lower accuracy, higher error, but not such a different behaviour between training and inference. Do you have any insights about that? $\endgroup$
    – Nerxis
    Commented Dec 19, 2018 at 10:42
  • $\begingroup$ @Nerxis I've personally seen instance norm to work quite well. Intuitively, as batch size goes to infinity, train and test time batch norm is the same. As batch size goes to 1, they become very different -- it's equivalent to using instance norm for train, and batch norm for test! $\endgroup$
    – shimao
    Commented Dec 21, 2018 at 1:31
  • $\begingroup$ Do you know if there is a good TensorFlow implementation of instance norm? I know about group norm but not about this particular one. $\endgroup$
    – Nerxis
    Commented Dec 21, 2018 at 9:02
  • $\begingroup$ @Nerxis tensorflow.org/api_docs/python/tf/contrib/layers/instance_norm $\endgroup$
    – shimao
    Commented Dec 21, 2018 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.