In some recent machine learning papers (e.g. mobileNetV2), ReLU6, defined as

$Relu(x)=\min(\max(0,x),6)$

is used instead of regular Relu non-linearities.

Doesn't such a function result in the same gradient vanishing problem as sigmoid functions? My assumption is that Batch normalization might be the reason that this works, but if that were true, then this would also hold for (scaled) sigmoid functions.

  • Please, do share the background of the magic number 6. Thank you – Jim Dec 6 at 18:43
  • @Jim To be honest I don't know. They say in the linked paper that this has to do with limiting accuracy loss when using low-precision arithmetic, but I haven't found anything precise on it (most of the justification seems to be experimental, although cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf seems to link it to something a bit more theoretical that I haven't had the time to go through.). Regardless, the gradient problem is what I find more puzzling at the moment... – Ash Dec 6 at 18:54
  • 1
    Batch normalized would also work for sigmoids. The issue is that sigmoids are 10x slower to compute compared to relus. So the speed advantage is considerable. – Alex R. Dec 6 at 22:48

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.