In the paper 'A Deep Non-Negative Matrix Factorization Neural Network' by Flunner and Hunter, proof of Theorem 1 says that "The ReLu Activation function is a standard approximation of a non-negative constraint", though no proof of such an argument is given, but I can't find such a claim anywhere, neither the experiments that I have run, shows confirmatory results. So, is the above argument in the paper true?

  • $\begingroup$ Not pretty sure the meaning of approximation here, but, "by construction", ReLU only outputs non-negative values; otherwise, clips the value to zero. $\endgroup$
    – hpwww
    Oct 17 '19 at 8:29
  • $\begingroup$ Yeah, that's true. But if my weights have been initialized as negative, the gradient of ReLU will be zero, hence not updating the weights, so the weights will remain negative, as my given input vectors are non negative. So, the "approximation" does not per se, hold if all my weights are initialized as negative. $\endgroup$ Oct 18 '19 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.