Rules for selecting convolutional neural network hyperparameters

Question

Are there any good papers that cover some methodical ways of picking the dimensions for filters, pooling the units, and determining the number of convolutional layers?

Answer 1

To some degree yes, a recent paper came out by Google researchers on how to choose good Inception architectures. Inception nets achieve very high performance on a constrained parameter budget, so this is as good of a place to start as any, and it's recent. Here's the link: Rethinking the Inception Architecture for Computer Vision.

They do not offer any hard quantitative rules but rather guidelines that they used and believe have helped them achieve good performance in the recent ImageNet competitions.

For example, some of the principles they discuss are:

• Use stacks of smaller receptive field convolutional layers instead of using a single large receptive field convolutional layers, i.e. 2 stacks of 3x3 conv layers vs a single 7x7 conv layer. This idea isn't new, it was also discussed in Return of the Devil in the Details: Delving Deep into Convolutional Networks by the Oxford VGG team. This is motivated by the need to be parameter efficient. It also has the dual effect of more representational capacity as we introduce more nonlinearity with more layers.

• Something that I haven't seen in the literature that this article mentioned is factorizing convolutional layers into deep layers. So instead of having a single 7x7 conv layer, we would have a 1x7 conv layer and then a 7x1 conv layer. Adds more depth, I believe it also parameter efficient as well.

• Balance the depth and width of your net. Use high dimensional representations. This is one of the principles behind their Inception modules, which concatenate multiple convolutinal layers together. So even if you have a small spatial size in your conv net, using Inception modules we can use a high dimensional representation via multi-scale convolutional concatenation: 1x1, 3x3, 3x3-3x3, max pool all put together. These Inception modules have a "width" since they can be interpreted as performing multiple operations in parallel. They go even further with new Inception modules which have factorized convolutional sizes, 1x3, 3x1, etc.

• Use 1x1 conv layers (Network in Network style) to reduce dimensionality. They use a lot of dimensionality reduction techniques to achieve parameter efficiency. They believe that this is effective because adjacent feature maps have highly correlated outputs. Which makes sense as natural images are known to exhibit some local statistical properties consistent with this. So reducing dimensionality via 1x1 NIN layers does not have a disastrous effect on representational power.

There's more in the article. I think it's an article that can offer some insight as to what you are asking about. They are talking about some very core concepts of conv net architectural design.

Answer 2

I haven't come across any literature on choosing these hyper-parameters as a function of the problem specifications. But, it's my understanding that most are adopting Bayesian optimization methods to zero in on effective values. You specify a reasonable range, and by testing various combinations, you learn a model of how those hyper-parameters relate to the model's accuracy. It's worked well for me. Check out "Practical Bayesian Optimization of Machine Learning Algorithms" from Snoek, Larochelle, and Adams (http://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf).