Are there simple networks where a ReLu between convolutional layers has significant value? At the moment I am studying the effect different non-linearities have on convolutional neural nets (CNNs). Since I'm not Google I am doing this by training simple nets (a few convolutional layers, followed by 1 or 2 fully connected layers, followed by softmax) on relatively simple datasets (MNIST, CIFAR-10).
Conventional wisdom says to


*

*add a ReLu after each layer (except before and after the softmax). 

*Do max-pooling after all (or most) convolutional layers.


According to the theory, the ReLu is crucial. This turns a mostly linear network into a network that could model basically any function given enough capacity in the network.
Because I want to research alternatives to the ReLu in CNNs, one thing I tried is to simply remove the ReLu in the convolutional layers. Then I can measure where alternatives lie on a scale of 'nothing' to 'relu'.
However to my surprise, removing the ReLu in the convolutional layers did almost nothing to the accuracy. Choosing average-pooling over max-pooling or removing the ReLu between fully connected layers (in the case of 2 fully-connected layers) was far more impactful.
Now my questions are:


*

*Are these findings consistent with what other people are getting?

*Does any one know a CNN architecture/dataset where removing the ReLu has a significant accuracy impact and which can be trained (lets say) within 24 hours with a sub 1000 dollar GPU? (training on Imagenet for each idea I want to test is not feasible for me)

*For my curiosity, is there any data how big of an impact removing the ReLus out of one of the big networks has?


update: what works is:
1. take the cifar-10 example of tensorflow.
2. Remove the ReLus in the fully-connected layers
3. Replace the max-pooling operations by average-pooling.
4. Now removing the ReLus in the convolutional layers lowers the accuracy from 83.4% to 37.2%.
 A: When using max pooling with relu, it might make sense that the difference between using relu and no activation function could be pretty small. To see why, consider all possible combinations during a 2x2 max pooling. If all 4 values are positive, they will all be left unchanged by relu, then pooling will choose the highest one, which will give the same result as using no activation function. If only one value is positive, the other 3 will be set to 0 by relu, then the only positive one will be chosen for max pooling, which would have been the same result as using no activation function. The only case where anything will happen differently is when all 4 values are negative. In this case, all 4 values would be set to 0 by relu, and the gradient for this area will be 0 during the backward pass. Without relu, it would choose the highest value and some gradient would be able to flow through. In the majority of cases, at least one value will be positive, and relu vs no relu will give identical results.  
This may very well be a hidden reason why relu is effective. One way to interpret all values in a certain NxN patch having negative values in a relu network is that there is nothing of use in that area, and relu will block the gradient from flowing through. This is all theory, but it might be that using relu allows the network to update the weights based only on areas that it finds useful, whereas other activation functions (or no activation function at all) will always update the weights based on every part of an image, even when most of the image is basically just noise.
