What are the rules of designing a good convolutional neural (such as VGG [1], Inception [2] Resnet[3], DenseNet[4] and so on...) network from scratch? What would be a recipe to start designing your own architecture?

Many meta-learning and architecture search algorithms (e.g, [5]) break-down the architecture design task into a simpler form. I wonder if a more general instruction can be gathered for researchers to follow.

I would appreciate it if you can provide a step-by-step guideline. Let's see if we can gather some useful tips and put them into test in here!

I will then follow these instructions and design a new architecture and will post the results in here.

[1] Very deep convolutional networks for large-scale image recognition K Simonyan, A Zisserman - arXiv preprint arXiv:1409.1556, 2014

[2] Szegedy, Christian, et al. "Inception-v4, inception-resnet and the impact of residual connections on learning." AAAI. Vol. 4. 2017.

[3] He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[4] Huang, Gao, et al. "Densely connected convolutional networks." Proceedings of the IEEE conference on computer vision and pattern recognition. Vol. 1. No. 2. 2017.

[5] Zoph, Barret, and Quoc V. Le. "Neural architecture search with reinforcement learning." arXiv preprint arXiv:1611.01578 (2016).

  • 1
    $\begingroup$ A golden rule that I'm fond of is that any golden rule is wrong. It's true that using $p = \sqrt{\text{number of features}}$ as the number of variables at each split of a random forest model is usually "good," but it's also true that cross-validation can usually find a better choice, and sometimes that choice is very far from $p$. Likewise, building a CNN for a particular task will require some amount of task-specific customization. See also: no free lunch theorem. $\endgroup$ – Sycorax says Reinstate Monica Apr 2 '18 at 15:43
  • 1
    $\begingroup$ "step-by-step guide" sounds like something that could be too broad for this site, could you make it more specific? $\endgroup$ – Tim Apr 2 '18 at 15:46
  • $\begingroup$ @Sycorax No free lunch indeed ...that's actually why I am asking this question. Searching randomly to come up with a good architecture for a new problem does not sound like a good idea. But perhaps there can be a recipe to design a successful architecture to follow... $\endgroup$ – PickleRick Apr 2 '18 at 15:48
  • 1
    $\begingroup$ "Deeper and more skip connections" sums up most of the recent work, I think. And a generous serving of tricks which help gradients flow better (batch norm, selu, etc). $\endgroup$ – shimao Apr 2 '18 at 15:55
  • 2
    $\begingroup$ My problem with this question is that any answer to it will most likely be outdated in a year's time. While architectures change all the time, it's also the intuition behind them that's perpetually changing. As an example, Resnets have been incredibly influential in the last few years, because '"residual connections are more efficient at deeper learning." But now we're learning that it's not the connections, as much as the niceness of gradients that result from those connections: arxiv.org/abs/1702.08591 $\endgroup$ – Alex R. Apr 2 '18 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.