I have read several papers on Convolutional Neural Nets but I am yet to come across any that has used thresholds on tanh or sigmoid to decide whether the neuron will fire or not. Obviously this works for ReLU, but why is it not used for tanh or sigmoid ? Having a 0.0001 coming out of an activation unit can do less good and much harm to a deep neural net because I am loosing sparsity.

On the other hand, I have encountered deep neural nets not able to use ReLU at all (specially in the fully connected layers) because it leads to explosion of gradients and un trainable nets. So I have to fall back on tanh and sigmoid.

Any advice will be really helpful Regards

  • $\begingroup$ Before choose your layer activation function you must check your data and in which format you will train it. tanh and sigmoid outputs are bounded value and rectifier output - is not. So, maybe rectifier works fine for you because your data is suited to this activation function? $\endgroup$ – itdxer Jan 29 '15 at 15:52
  • $\begingroup$ Well yes - tanh and sigmoid works for me - but the results are not so good - may be because I am loosing the sparsity (regularization) from the sigmoid outputs ? That was my question - as in - has anyone come across such thresholds in literature - OR - is there any concrete logic why Conv Neural Nets do not use thresholds for non linearity ? $\endgroup$ – Run2 Jan 30 '15 at 4:55
  • $\begingroup$ So, as I say rectifier layer has no upper bound, so for prediction of not bounded values it's give you better result. For example: you have some data like price of somethid, this value have no upper bound and you choose sigmoid as a layer output, so you will get data between 0 and 1 and every output between 1 and inf will be always with error which you can't minimize. $\endgroup$ – itdxer Jan 30 '15 at 11:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.