In convolutional neural networks (CNNs) it is common to intersperse convolutional layers with non-linearities, local-response-normalizations and possibly pooling-layers. In the literature I found plenty discussion of when and why to use these different layers. However when using 2 or more of these layers between 2 convolutions, I've seen no arguments on what order to use them. This usually means either: it doesn't matter, or that there is an obvious way to order them or that this is part of the black magic that makes the networks work best that nobody talks about. Can anyone give me some guidance on what order to use them?
Context: I started thinking about this when I saw that the inference function of the tensorflow cifar-10 example network use 2 different orderings after its 2 convolutional layers: first relu-pool-normalization and then relu-normalization-pool.
