My physics professors in grad school, as well as the Noble laureate Feynman, would always present what they called toy models to illustrate basic concepts and methods in physics, such as the harmonic oscillator, pendulum, spinning top, and black box.

What toy models are used to illustrate the basic concepts and methods underlying the application of neural networks? (References please.)

By a toy model I mean a particularly simple, minimally sized network applied to a highly constrained problem through which basic methods can be presented and one's understanding tested and enhanced through actual implementation, i.e., constructing the basic code and preferably to a certain degree doing/checking the basic math by hand or aided by a symbolic math app.


One of the most classical is the Perceptron in 2 dimensions, which goes back to the 1950s. This is a good example because it is a launching pad for more modern techniques:

1) Not everything is linearly separable (hence the need for nonlinear activations or kernel methods, multiple layers, etc.).

2) The Perceptron won't converge if the data is not linearly separable (continuous measures of separation such as softmax, learning rate decay, etc.).

3) While there are infinitely many solutions to splitting data, it's clear that some are more desired than others (maximum boundary separation, SVMs, etc.)

For multilayer neural networks, you might like the toy classification examples that come with this visualization.

For Convolutional Neural Nets, the MNIST is the classical gold standard, with a cute visualization here and here.

For RNNs, a really simple problem they can solve is binary addition, which requires memorizing 4 patterns.

| cite | improve this answer | |
  • $\begingroup$ +1 for the wide coverage of NN! from perceptron to RNN. $\endgroup$ – Haitao Du May 15 '17 at 16:10
  • $\begingroup$ Nice. The type of answers I'm looking for. $\endgroup$ – Tom Copeland May 15 '17 at 16:23
  • $\begingroup$ Sorry, I didn't mean to edit your answer -- I meant to add that paragraph to mine. $\endgroup$ – Sycorax May 15 '17 at 18:09
  1. The XOR problem is probably the canonical ANN toy problem.

    Richard Bland June 1998 University of Stirling, Department of Computing Science and Mathematics Computing Science Technical Report "Learning XOR: exploring the space of a classic problem"

  2. The TensorFlow Playground is an interactive interface to several toy neural nets, including XOR and Jellyroll.

  3. Computing the largest eigenvalue of a fixed-size (2x2 or 3x3) symmetric matrix is one I use in classroom demos.

    A. Cichocki and R. Unbehauen. "Neural networks for computing eigenvalues and eigenvectors" Biological Cybernetics December 1992, Volume 68, Issue 2, pp 155–164

Problems like MNIST are definitely canonical but aren't easily verified by hand -- unless you happen to have enormous free time. Nor is the code especially basic.

As far as NLP tasks, the Penn Tree Bank is a very standard benchmark data set, used for example in Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals "Recurrent Neural Network Regularization," and probably hundreds of other papers.

| cite | improve this answer | |

I don't know of a physical toy, but the best example I know of is a multi-layer AI generated through a genetic algorithm to play Super Mario Brothers. The source code is in the video description.

MarI/O - Machine Learning for Video Games: https://www.youtube.com/watch?v=qv6UVOQ0F44

| cite | improve this answer | |
  • 3
    $\begingroup$ You might want to thoroughly read the question and the other two answers. $\endgroup$ – Tom Copeland May 15 '17 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.