Neural architectures: data-informed automatic design

Question

The recent progress in neural networks is summarized by a sequence of novel architectures characterized mainly by its growing design complexity. From LeNet5 (1994) to AlexNet (2012), to Overfeat (2013) and GoogleLeNet/Inception (2014) and so on...

Is there any attempt to let the machine decide/design which architecture to be used depending on the data?

Answer 1

You may also want to look at the literature on neuro-evolution. Examples:

• Zaremba, Wojciech. Ilya Sutskever. Rafal Jozefowicz "An empirical exploration of recurrent network architectures." (2015): used evolutionary computation to find optimal RNN structures.
• Franck Dernoncourt. "The medial Reticular Formation: a neural substrate for action selection? An evaluation via evolutionary computation.". Master's Thesis. École Normale Supérieure Ulm. 2011.
• Bayer, Justin, Daan Wierstra, Julian Togelius, and Jürgen Schmidhuber. "Evolving memory cell structures for sequence learning." In International Conference on Artificial Neural Networks, pp. 755-764. Springer Berlin Heidelberg, 2009.: used evolutionary computation to find optimal RNN structures.

Reinforcement learning:

• Pham, Hieu, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. "Efficient Neural Architecture Search via Parameter Sharing." arXiv preprint arXiv:1802.03268 (2018). https://arxiv.org/pdf/1802.03268.pdf
• Zoph, Barret and Le, Quoc V. Neural architecture search with reinforcement learning. In ICLR, 2017. https://arxiv.org/abs/1611.01578
• Jose M Alvarez, Mathieu Salzmann. Learning the Number of Neurons in Deep Networks. NIPS 2016. https://arxiv.org/abs/1611.06321

• Bowen Baker, Otkrist Gupta, Nikhil Naik, Ramesh Raskar. Designing Neural Network Architectures using Reinforcement Learning. https://arxiv.org/abs/1611.02167

• Barret Zoph, Quoc V. Le. Neural Architecture Search with Reinforcement Learning. https://arxiv.org/abs/1611.01578

Miscellaneous:

• Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Nando de Freitas. Learning to learn by gradient descent by gradient descent. https://arxiv.org/abs/1606.04474
• Franck Dernoncourt, Ji Young Lee Optimizing Neural Network Hyperparameters with Gaussian Processes for Dialog Act Classification, IEEE SLT 2016.
• Cortes, Corinna, Xavi Gonzalvo, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. "AdaNet: Adaptive Structural Learning of Artificial Neural Networks." arXiv preprint arXiv:1607.01097 (2016). https://arxiv.org/abs/1607.01097 : Approach that learns both the structure of the network as well as its weights.

Answer 2

It seems research is moving towards such direction:

• Miconi, Thomas. "Neural networks with differentiable structure.", arXiv (2016): the author introduce the concept of differentiable network structure to include it in the objective function.

Answer 3

Google's AutoML is one such example. It was successfully applied on the ImageNet dataset, resulting NASNet, which as of this writing, has surpassed all other models in terms of accuracy. The main paper is here:

Learning Transferable Architectures for Scalable Image Recognition -- Zoph, et. al.