2
$\begingroup$

So let's say I want to train a neural network for a classification task. It is somehow logical that there should exist a way to pick training examples in a really smart way so that the training can be done with smaller number of examples (hence more efficiently).

E.g. a couple of different examples can influence the weight changes in the same way as one example which would then "encompass" all these different examples (parallel to generating corner-case examples while testing software).

Can this reasoning even be applied to neural nets? If so, is there any work on this? I can't seem to find any, which might very well be because training neural nets simply can't be thought of in this way). However I did read about one-shot learning, is this somehow related to what I explained and should I investigate more about it? I really hope that there is some other way to do training than just throwing as many examples as possible at the network.

Thank you very much!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ There is never a real substitute for more data. However, there are several techniques for training better, given the data that you have. Bootstrapping and multiplying your dataset through transformations (such as rotation and translation of images, or adding noise). I believe that one-shot learning is often used to re-train an existing system. I don't think that it is a substitute for scale of the original dataset. Also, if you have a small dataset, correctly using regularization can be important to avoid over fitting to peculiarities of the small dataset. $\endgroup$
    – John Yetter
    Sep 8, 2016 at 13:37
  • 1
    $\begingroup$ Aside from the experimental design aspect, another angle is labeled vs. unlabeled examples. The talk Learning Feature Hierarchies (LeCun 2009) focuses on leveraging a large dataset of unlabeled examples to allow improved supervised learning results on a much smaller dataset of labeled examples. As a motivating example he mentions SIFT, which encodes a lot of "prior information" to perform well on Caltech 101. $\endgroup$
    – GeoMatt22
    Sep 8, 2016 at 17:05

2 Answers 2

Reset to default
2
$\begingroup$

Yes, in machine learning this would be called "active learning", where samples are iteratively added to the dataset in order to minimise the uncertainty of the model (in some sense). ISTR that one of David MacKay's early papers (arising from his PhD) on Bayesian neural networks had some coverage of this (866 Google scholar citations suggests it is a good place to start):

David J. C. MacKay, "Information-Based Objective Functions for Active Data Selection", Neural Computation, Vol. 4, No. 4, Pages 590-604, July 1992. (doi:10.1162/neco.1992.4.4.590)

Improve this answer
$\endgroup$
2
$\begingroup$

I agree with Dikran's answer and I would like to add a couple of observations to it that might be constructive/relevant.

  1. From an engineering perspective, Active learning within machine learning has an analog within engineering, which is "sequential design of experiments". Both fields strive to sequentially suggest additional training locations that will best maximize information gain about the statistical model's parameters. In fact, Mackay's work in 1992 was inspired by ideas from optimal experimental design. So, basically one should have a basic understanding about the theory of optimal experimental design to address such problem.

  2. A nice and broader presentation on how to project ideas from experimental design to machine learning, in order to suggest the "optimal" training examples, can be found in a later paper by David Cohn in 1996, that extended the earlier work of Mackay:

David A. Cohn, Zoubin Ghahramani and Michael Jordan, "Active Learning with Statistical Models," Journal of Artificial Intelligence Research 4, 1996, 129-145.

Improve this answer
$\endgroup$
1
  • $\begingroup$ +1 Burr Settles survey of active learning is also a good place to look for more info. $\endgroup$
    – Dikran Marsupial
    Sep 12, 2016 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.