At the moment I am studying the effect different non-linearities have on convolutional neural nets (CNNs). Since I'm not Google I am doing this by training simple nets (a few convolutional layers, followed by 1 or 2 fully connected layers, followed by softmax) on relatively simple datasets (MNIST, CIFAR-10).
Conventional wisdom says to
- add a ReLu after each layer (except before and after the softmax).
- Do max-pooling after all (or most) convolutional layers.
According to the theory, the ReLu is crucial. This turns a mostly linear network into a network that could model basically any function given enough capacity in the network.
Because I want to research alternatives to the ReLu in CNNs, one thing I tried is to simply remove the ReLu in the convolutional layers. Then I can measure where alternatives lie on a scale of 'nothing' to 'relu'.
However to my surprise, removing the ReLu in the convolutional layers did almost nothing to the accuracy. Choosing average-pooling over max-pooling or removing the ReLu between fully connected layers (in the case of 2 fully-connected layers) was far more impactful.
Now my questions are:
- Are these findings consistent with what other people are getting?
- Does any one know a CNN architecture/dataset where removing the ReLu has a significant accuracy impact and which can be trained (lets say) within 24 hours with a sub 1000 dollar GPU? (training on Imagenet for each idea I want to test is not feasible for me)
- For my curiosity, is there any data how big of an impact removing the ReLus out of one of the big networks has?
update: what works is:
1. take the cifar-10 example of tensorflow.
2. Remove the ReLus in the fully-connected layers
3. Replace the max-pooling operations by average-pooling.
4. Now removing the ReLus in the convolutional layers lowers the accuracy from 83.4% to 37.2%.