# ReLU for small artificial neural networks? [duplicate]

This question already has an answer here:

I know how effective ReLU are for deep neural networks, however i wanted to know if they can be used for smaller networks as well ? ( like a network with 2 hidden layers and 15 units each ).

## marked as duplicate by Sycorax neural-networks StackExchange.ready(function() { if (StackExchange.options.isMobile) return; $('.dupe-hammer-message-hover:not(.hover-bound)').each(function() { var$hover = $(this).addClass('hover-bound'),$msg = $hover.siblings('.dupe-hammer-message');$hover.hover( function() { $hover.showInfoMessage('', { messageElement:$msg.clone().show(), transient: false, position: { my: 'bottom left', at: 'top center', offsetTop: -7 }, dismissable: false, relativeToBody: true }); }, function() { StackExchange.helpers.removeMessages(); } ); }); }); May 16 at 3:03

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

• Why not? Activation layer is just a transformation. As long as your network itself is a good fit for your data set, everything should be ok. – SmallChess Sep 20 '16 at 7:16
• Same question – Asciiom Sep 20 '16 at 8:55
• I found it not performing well in super small tiny networks, but maybe I'm making some error. Please refer: stats.stackexchange.com/questions/284203/… – thedk Jun 8 '17 at 22:27

## 1 Answer

Can be used, yes. Are effective depends on your network, on your problem, and compared to what other activation function. In my experience ReLUs are pretty good even in small networks on classical problems (MNIST, CIFAR10) compared to tanh or sigmoid. The difference is more obvious in deeper networks though.

• I think the answer is wrong. There's evidence that ReLU is also good for deep networks. In fact I believe Google trained a 17-layer network with ReLu for AlphaGo. – SmallChess Sep 20 '16 at 9:07
• My formulation might be a little bit wonky, edited. I did mean to say that they perform even better in deeper network. – Jenkar Sep 20 '16 at 9:11