Minimizing the number of examples needed for training a neural network? So let's say I want to train a neural network for a classification task. It is somehow logical that there should exist a way to pick training examples in a really smart way so that the training can be done with smaller number of examples (hence more efficiently).
E.g. a couple of different examples can influence the weight changes in the same way as one example which would then "encompass" all these different examples (parallel to generating corner-case examples while testing software).
Can this reasoning even be applied to neural nets? If so, is there any work on this? I can't seem to find any, which might very well be because training neural nets simply can't be thought of in this way). However I did read about one-shot learning, is this somehow related to what I explained and should I investigate more about it? I really hope that there is some other way to do training than just throwing as many examples as possible at the network.
Thank you very much!
 A: Yes, in machine learning this would be called "active learning", where samples are iteratively added to the dataset in order to minimise the uncertainty of the model (in some sense).  ISTR that one of David MacKay's early papers (arising from his PhD) on Bayesian neural networks had some coverage of this (866 Google scholar citations suggests it is a good place to start):
David J. C. MacKay, "Information-Based Objective Functions for Active Data Selection", Neural Computation, Vol. 4, No. 4, Pages 590-604, July 1992.
(doi:10.1162/neco.1992.4.4.590) 
A: I agree with Dikran's answer and I would like to add a couple of observations to it that might be constructive/relevant. 


*

*From an engineering perspective, Active learning within machine learning has an analog within engineering, which is "sequential design of experiments". Both fields strive to sequentially suggest additional training locations that will best maximize information gain about the statistical model's parameters. In fact, Mackay's work in 1992 was inspired by ideas from optimal experimental design. So, basically one should have a basic understanding about the theory of optimal experimental design to address such problem. 

*A nice and broader presentation on how to project ideas from experimental design to machine learning, in order to suggest the "optimal" training examples, can be found in a later paper by David Cohn in 1996, that extended the earlier work of Mackay:
David A. Cohn, Zoubin Ghahramani and Michael Jordan, "Active Learning with Statistical Models," Journal of Artificial Intelligence Research 4, 1996, 129-145. 
