Often I have heard the data miners here use this term. As a statistician who has worked on classification problems I am familiar with the term "train a classifier" and I assume "learn a model" means the same thing. I don't mind the term "train a classifier". That seems to portray the idea of fitting a model as the training data is used to get good or "improved" estimates of the model parameters. But the would learn means to gain knowledge. In plain English "learn a model" would mean to know what it is. But in fact we never "know" the model. Models approximate reality but no model is correct. It is like Box said "No model is correct but some are useful."

I would be interested to hear the data miners response. How did the term originate? If you use it, why do you like it?

  • $\begingroup$ The funny part is that "learn a model" means at most "memorize all its parameters" or so, thus it is incorrect even if you won't argue whether "learning" is a good word here. In my language (pl), the official nomenclature is to "teach a model", although "machine learning" is called "machine learning" rather than "machine teaching". $\endgroup$ – user88 Sep 10 '12 at 13:11
  • $\begingroup$ Sidebar: Do you have a reference for the Box quotation. $\endgroup$ – entropy Jan 2 '13 at 22:56

I suspect its origins are in the artificial neural network research community, where the neural network can be thought of as learning a model of the data via modification of synaptic weights in a similar manner to that which occurs in the human brain as we ourselves learn from experience. My research career started out in artificial neural networks so I sometimes use the phrase.

Perhaps it makes more sense if you think of the model as being encoded in the parameters of the model, rather than the equation, in the same way that a mental model is not an identifiable physical component of the brain as much as a set of parameter settings for some of our neurons.

Note that there is no implication that a mental model is necessarily correct either!

  • 1
    $\begingroup$ My study leader and I had this discussion, and I posted a question related to this discussion here stats.stackexchange.com/questions/43559/…. I came to understand that model fitting, training, estimation and learning just mean the application of some optimization method to find the best parameters that defines a model of a specific form. $\endgroup$ – entropy Jan 2 '13 at 22:59

The term is quite old in artifical intelligence. Turing devoted a long section on "Learning Machines" in his 1950 Computing Machinery and Intelligence paper in Mind, and qualitatively sketches out supervised learning. Rosenblatt's original paper: The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain paper from 1958 talks extensively about a "Mathematical Model of Learning". Here the perceptron was a "model of learning"; models were not "learned".

The Pitts and McCullough 1943 paper - the original "neural networks" paper - wasn't really concerned with learning, more how one could construct a logical calculus (like a Hilbert or Gentzen system, but I think they refer to Russell/Whitehead) which could perform inference. I think it was the "Perceptrons" paper which introduced a numerical, as opposed to symbolic notion of learning in this tradition.

Is it possible for a machine to learn how to play chess just from examples? Yes. Does it have a model for chess playing? Yes. Is it the optimal model (assuming there is one)? Almost certainly not. In plain English I've "learned chess" if I can play chess ok - or maybe pretty well. It doesn't mean I'm the optimal chess player. This is the sense in which Turing was describing "learning" when he discussed learning chess in his paper.

I'm very inconsistent with what term I use. So (for instance) for learning-in-the-limit I'd say "identify", for SVM-learning I'd say "train", but for MCMC-"learning" I'd say "optimize". And e.g. I just call regression "regression".

  • $\begingroup$ My question was not about using the term learning in classification but rather the phrase "learning a model". Machine learning is very familiar but "learning a model" is a phrase I heard for the first time here on this site. $\endgroup$ – Michael Chernick Sep 11 '12 at 0:52
  • $\begingroup$ I've added a little blurb there, but you're right, this isn't a great answer, I've hammered on learning without much discussion of "model". I'll see if I can remember anything today, but I might remove it this evening if nothing comes to mind. $\endgroup$ – Patrick Caldon Sep 11 '12 at 1:01
  • $\begingroup$ I was going to delete this answer, but clearly a few people like it! I guess it's a reasonable little synopsis of where the term "learning" comes from in this context. $\endgroup$ – Patrick Caldon Sep 17 '12 at 4:56
  • $\begingroup$ +1 I am giving an upvote because although a little off the point of the question it is related and interesting and did get 3 upvotes. Keep it in. $\endgroup$ – Michael Chernick Sep 17 '12 at 10:50

As a researcher in Bioplausible Machine Learning, I strongly agree that "no model is correct but some are useful", and in fact models and formalisms have the strong failing as used by authors who talk about optimization of the problem, when what they are doing is optimizing a model, i.e. exploring its parameter space and finding a local or hopefully global optimum. This is not in general an optimum for the real problem. While the originator of a model normally uses the correct terminology, and exposes all the assumptions, most users gloss over the assumptions, which most often are known not to hold, and also use less precise language about "learning" and "optimization" and "parameterization".

I think this optimal parameterization of a model is what people would mean in Machine Learning, particularly in supervised Machine Learning, although I can't say I've heard "learn a model" a lot - but it does occur, and whereas the person trains the model, the computer learns the parameters of the model. Even in unsupervised learning the "learning" is most often simply parameterization of a model, and hopefully "learning a model" is thus optimal parameterization of a model (although often different ways of searching the parameter space find different solutions even though they can be proven to optimize the same thing). I would indeed rather use "training a model" for this purpose rather than personifying the computer and creating the ambiguity between discovering a new model and parameterizing an old one.

In fact, most of my research is about learning the model in terms of discovering a better model, or a more computationally and cognitively/biologically/ecologically plausible model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.