Where did the term "learn a model" come from

Question

Often I have heard the data miners here use this term. As a statistician who has worked on classification problems I am familiar with the term "train a classifier" and I assume "learn a model" means the same thing. I don't mind the term "train a classifier". That seems to portray the idea of fitting a model as the training data is used to get good or "improved" estimates of the model parameters. But the would learn means to gain knowledge. In plain English "learn a model" would mean to know what it is. But in fact we never "know" the model. Models approximate reality but no model is correct. It is like Box said "No model is correct but some are useful."

I would be interested to hear the data miners response. How did the term originate? If you use it, why do you like it?

Answer 1

I suspect its origins are in the artificial neural network research community, where the neural network can be thought of as learning a model of the data via modification of synaptic weights in a similar manner to that which occurs in the human brain as we ourselves learn from experience. My research career started out in artificial neural networks so I sometimes use the phrase.

Perhaps it makes more sense if you think of the model as being encoded in the parameters of the model, rather than the equation, in the same way that a mental model is not an identifiable physical component of the brain as much as a set of parameter settings for some of our neurons.

Note that there is no implication that a mental model is necessarily correct either!

Answer 2

The term is quite old in artifical intelligence. Turing devoted a long section on "Learning Machines" in his 1950 Computing Machinery and Intelligence paper in Mind, and qualitatively sketches out supervised learning. Rosenblatt's original paper: The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain paper from 1958 talks extensively about a "Mathematical Model of Learning". Here the perceptron was a "model of learning"; models were not "learned".

The Pitts and McCullough 1943 paper - the original "neural networks" paper - wasn't really concerned with learning, more how one could construct a logical calculus (like a Hilbert or Gentzen system, but I think they refer to Russell/Whitehead) which could perform inference. I think it was the "Perceptrons" paper which introduced a numerical, as opposed to symbolic notion of learning in this tradition.

Is it possible for a machine to learn how to play chess just from examples? Yes. Does it have a model for chess playing? Yes. Is it the optimal model (assuming there is one)? Almost certainly not. In plain English I've "learned chess" if I can play chess ok - or maybe pretty well. It doesn't mean I'm the optimal chess player. This is the sense in which Turing was describing "learning" when he discussed learning chess in his paper.

I'm very inconsistent with what term I use. So (for instance) for learning-in-the-limit I'd say "identify", for SVM-learning I'd say "train", but for MCMC-"learning" I'd say "optimize". And e.g. I just call regression "regression".

Answer 3

As a researcher in Bioplausible Machine Learning, I strongly agree that "no model is correct but some are useful", and in fact models and formalisms have the strong failing as used by authors who talk about optimization of the problem, when what they are doing is optimizing a model, i.e. exploring its parameter space and finding a local or hopefully global optimum. This is not in general an optimum for the real problem. While the originator of a model normally uses the correct terminology, and exposes all the assumptions, most users gloss over the assumptions, which most often are known not to hold, and also use less precise language about "learning" and "optimization" and "parameterization".

I think this optimal parameterization of a model is what people would mean in Machine Learning, particularly in supervised Machine Learning, although I can't say I've heard "learn a model" a lot - but it does occur, and whereas the person trains the model, the computer learns the parameters of the model. Even in unsupervised learning the "learning" is most often simply parameterization of a model, and hopefully "learning a model" is thus optimal parameterization of a model (although often different ways of searching the parameter space find different solutions even though they can be proven to optimize the same thing). I would indeed rather use "training a model" for this purpose rather than personifying the computer and creating the ambiguity between discovering a new model and parameterizing an old one.

In fact, most of my research is about learning the model in terms of discovering a better model, or a more computationally and cognitively/biologically/ecologically plausible model.