In some recent machine learning papers (e.g. mobileNetV2), ReLU6, defined as


is used instead of regular Relu non-linearities.

Doesn't such a function result in the same gradient vanishing problem as sigmoid functions? My assumption is that Batch normalization might be the reason that this works, but if that were true, then this would also hold for (scaled) sigmoid functions.

  • 1
    $\begingroup$ Please, do share the background of the magic number 6. Thank you $\endgroup$
    – Jim
    Dec 6 '18 at 18:43
  • $\begingroup$ @Jim To be honest I don't know. They say in the linked paper that this has to do with limiting accuracy loss when using low-precision arithmetic, but I haven't found anything precise on it (most of the justification seems to be experimental, although cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf seems to link it to something a bit more theoretical that I haven't had the time to go through.). Regardless, the gradient problem is what I find more puzzling at the moment... $\endgroup$
    – Ash
    Dec 6 '18 at 18:54
  • 3
    $\begingroup$ Batch normalized would also work for sigmoids. The issue is that sigmoids are 10x slower to compute compared to relus. So the speed advantage is considerable. $\endgroup$
    – Alex R.
    Dec 6 '18 at 22:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.