19
$\begingroup$

Machine learning research papers often treat learning and inference as two separate tasks, but it is not quite clear to me what the distinction is. In this book for example they use Bayesian statistics for both kinds of tasks, but do not provide a motivation for that distinction. I have several vague ideas what it could be about, but I would like to see a solid definition and perhaps also rebuttals or extensions of my ideas:

  • The difference between inferring the values of latent variables for a certain data point, and learning a suitable model for the data.
  • The difference between extracting variances (inference) and learning the invariances so as to be able to extract variances (by learning the dynamics of the input space/process/world).
  • The neuroscientific analogy might be short-term potentiation/depression (memory traces) vs long-term potentiation/depression.
$\endgroup$
  • 4
    $\begingroup$ Not sure whether this helps, but in statistics one distinction is between whether you want to think about learning as inference (mostly Bayes) or as estimation (mostly Frequentist). For the former, learning about everything - latent variables, parameters, predictions, models - is an inference (which returns a distribution). For the latter some learning problems may be an inference and others an estimation problem (which returns an estimate and sampling-theoretically motivated uncertainty range for it). $\endgroup$ – conjugateprior Apr 3 '16 at 17:16
  • 5
    $\begingroup$ "Learning" is just an evocative metaphor for the process of training a machine learning algorithm. I don't think there's much insight to be gained here. $\endgroup$ – Sycorax Apr 3 '16 at 17:40
  • 1
    $\begingroup$ Possible duplicate of The Two Cultures: statistics vs. machine learning? $\endgroup$ – Winks Apr 3 '16 at 19:05
  • 1
    $\begingroup$ @Winks Did you read the linked question at all? None of the answers makes the distinction I am asking for explicit. $\endgroup$ – Lenar Hoyt Apr 3 '16 at 19:20
  • 1
    $\begingroup$ @conjugateprior In machine learning, no one would say that "learning about everything - latent variables, parameters, predictions, models - is an inference". Learning and inference are considered totally separate even though they both can produce distributions. $\endgroup$ – Neil G Apr 3 '16 at 19:42
11
$\begingroup$

I agree with Neil G's answer, but perhaps this alternative phrasing also helps:

Consider the setting of a simple Gaussian mixture model. Here we can think of the model parameters as the set of Gaussian components of the mixture model (each of their means and variances, and each one's weight in the mixture).

Given a set of model parameters, inference is the problem of identifying which component was likely to have generated a single given example, usually in the form of a "responsibility" for each component. Here, the latent variables are just the single identifier for which component generated the given vector, and we are inferring which component that was likely to have been. (In this case, inference is simple, though in more complex models it becomes quite complicated.)

Learning is the process of, given a set of samples from the model, identifying the model parameters (or a distribution over model parameters) that best fit the data given: choosing the Gaussians' means, variances, and weightings.

The Expectation-Maximization learning algorithm can be thought of as performing inference for the training set, then learning the best parameters given that inference, then repeating. Inference is often used in the learning process in this way, but it is also of independent interest, e.g. to choose which component generated a given data point in a Gaussian mixture model, to decide on the most likely hidden state in a hidden Markov model, to impute missing values in a more general graphical model, ....

$\endgroup$
  • 1
    $\begingroup$ And a small caveat that one can choose to break things down into learning and inference this way, but one can also choose do the whole lot as inference: stats.stackexchange.com/questions/180582/… $\endgroup$ – conjugateprior Apr 3 '16 at 22:15
  • $\begingroup$ Why so many lines? I want to see a simple answer which differentiates them in one-two sentences. Also, not everyone is familiar with GMMs or EM. $\endgroup$ – nbro Jan 19 at 12:26
9
$\begingroup$

Inference is choosing a configuration based on a single input. Learning is choosing parameters based on some training examples.

In the energy-based model framework (a way of looking at nearly all machine learning architectures), inference chooses a configuration to minimize an energy function while holding the parameters fixed; learning chooses the parameters to minimize the loss function.

As conjugateprior points out, other people use different terminology for the same thing. For example Bishop, uses "inference" and "decision" to mean learning and inference respectively. Causal inference means learning. But whichever terms you decide on, these two concepts are distinct.

The neurological analogy is a pattern of firing neurons is a configuration; a set of link strengths are the parameters.

$\endgroup$
  • $\begingroup$ @mcb I still don't know what you mean by "variances". "Invariances" isn't even a word in the dictionary. Yes, there are many learning algorithms that rely on an inferred configuration like EM described in Dougal's answer. $\endgroup$ – Neil G Apr 3 '16 at 19:36
  • $\begingroup$ @mcb I don't understand your questions either; perhaps it would help to specify an example model and be specific about what distribution / variances / invariants(?) you're talking about. $\endgroup$ – Dougal Apr 3 '16 at 20:03
  • $\begingroup$ Thanks for Your answers. Perhaps I have misunderstood something. $\endgroup$ – Lenar Hoyt Apr 3 '16 at 21:20
  • $\begingroup$ @NeilG I believe this terminology is mostly used in ML vision work where classification decisions should be 'invariant' to object translation, rotation, rescaling etc. Can't find a good short reference, but there's this: en.wikipedia.org/wiki/Prior_knowledge_for_pattern_recognition $\endgroup$ – conjugateprior Apr 3 '16 at 22:20
  • $\begingroup$ @conjugateprior I had a feeling that's what he was getting at, but I wanted to see if he would make his question clear. $\endgroup$ – Neil G Apr 3 '16 at 22:22
4
$\begingroup$

This looks like classic cross-discipline lingo confusion. The OP seems to be using neuroscience-like terminology where the two terms in question may have different connotations. But since Cross Validated generally deals with statistics and maching learning, I'll try answering the question based on the common usage of these terms in those fields.

In classical statistics, inference is simply the act of taking what you know about a sample and making a mathematical statement about the population from which it is (hopefully) representative. From the canonical textbook of Casella & Berger (2002): "The subject of probability theory is the foundation upon which all of statistics is built ... through these models, statisticians are able to draw inferences about populations, inferences based on examination of only a part of the whole". So in statistics, inference is specifically related to p-values, test statistics, and sampling distributions, etc.

As for learning, I think this table from Wasserman's All of Statistics (2003) might be helpful:

enter image description here

$\endgroup$
  • $\begingroup$ This disagrees with a lot of other textbooks including Bishop's book mentioned in the comments. Classification is a kind of supervised learning when the target variables are categories. The word "estimation" alone is vague: usually we mean "density estimation" or "parameter estimation" or "sequential estimation" or "maximum likelihood estimation". $\endgroup$ – Neil G Apr 3 '16 at 22:11
  • 1
    $\begingroup$ Also, Bayes net is not just a directed acyclic graph! It is a kind of dag whose nodes represent propositions and whose edges represent probabilistic dependencies. It specifies conditional independence relationships. $\endgroup$ – Neil G Apr 3 '16 at 22:15
  • 1
    $\begingroup$ @NeilG Quite so. The closest statistics translation would probably be "structural equation model" $\endgroup$ – conjugateprior Apr 3 '16 at 22:50
  • 2
    $\begingroup$ And in a dismaying amount of statistics there should be two lines about data: CS: training data, Statistics: data. CS: test data, Statistics: wut? $\endgroup$ – conjugateprior Apr 3 '16 at 22:52
  • $\begingroup$ Stat 101: wut = another (hopefully random) sample from your population ... $\endgroup$ – Zoë Clark Apr 4 '16 at 1:36
-1
$\begingroup$

It is strange no one else mentioned this, but you can have inference only in cases where you have a probability distribution. Here to quote Wiki, which quotes Oxford dictionary:

Statistical inference is the process of using data analysis to deduce properties of an underlying probability distribution (Oxford Dictionary of Statistics)

https://en.wikipedia.org/wiki/Statistical_inference

In case of traditional neural networks, k-NN or vanilla SVMs you have no probability density to estimate, nor assumptions about any density, thus, no statistical inference there. Only training/learning. However, for most (all?) statistical procedures, you can use both inference AND learning, since these procedures possess some assumptions about the distribution of population in question.

$\endgroup$
  • $\begingroup$ This is wrong. Anyway, you can interpret neural networks as producing a distribution if you want. See, e.g. Amari 1998. $\endgroup$ – Neil G Jan 19 at 16:44
  • $\begingroup$ It is not wrong, or specify. You CAN interpret, but originally there is no such interpretation. $\endgroup$ – SWIM S. Jan 19 at 16:52
  • $\begingroup$ It is wrong because people use the term inference with models like autoenciders. $\endgroup$ – Neil G Jan 19 at 16:58
  • $\begingroup$ So, is it wrong because some group of people uses the term incorrectly? Or because they have some probabilistic interpretation for their NNs (i'm not deeply familiar with autoencoders)? I logically justified why one term is different from the other. So, given the definition above, i see that those who use the term inference with NNs, k-NNs, or SVMs (unless with probabilistic interpretation) are pretty much abusing the notation. $\endgroup$ – SWIM S. Jan 19 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.