0
$\begingroup$

This question already has an answer here:

I know how effective ReLU are for deep neural networks, however i wanted to know if they can be used for smaller networks as well ? ( like a network with 2 hidden layers and 15 units each ).

$\endgroup$

marked as duplicate by Sycorax neural-networks May 16 at 3:03

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ Why not? Activation layer is just a transformation. As long as your network itself is a good fit for your data set, everything should be ok. $\endgroup$ – SmallChess Sep 20 '16 at 7:16
  • $\begingroup$ Same question $\endgroup$ – Asciiom Sep 20 '16 at 8:55
  • $\begingroup$ I found it not performing well in super small tiny networks, but maybe I'm making some error. Please refer: stats.stackexchange.com/questions/284203/… $\endgroup$ – thedk Jun 8 '17 at 22:27
0
$\begingroup$

Can be used, yes. Are effective depends on your network, on your problem, and compared to what other activation function. In my experience ReLUs are pretty good even in small networks on classical problems (MNIST, CIFAR10) compared to tanh or sigmoid. The difference is more obvious in deeper networks though.

$\endgroup$
  • $\begingroup$ I think the answer is wrong. There's evidence that ReLU is also good for deep networks. In fact I believe Google trained a 17-layer network with ReLu for AlphaGo. $\endgroup$ – SmallChess Sep 20 '16 at 9:07
  • $\begingroup$ My formulation might be a little bit wonky, edited. I did mean to say that they perform even better in deeper network. $\endgroup$ – Jenkar Sep 20 '16 at 9:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.