# What is probabilistic inference?

I am reading Chris Bishop's Pattern Recognition and Machine Learning textbook. I came across the term probabilistic inference several times. I have a couple of questions.

1. Is probabilistic inference only applicable in a graphical modelling context?

2. What's the distinction between traditional statistical inference (p-values, confidence intervals, Bayes factors etc.) and probabilistic inference?

3. Is this a term that's specific to the CS community or is it widely used in the statistics community as well?

• In my opinion, this is just a fancy denomination (and an oxymoron) that reproduces the fact that statistics is based on probabilistic modelling. – Xi'an Nov 2 '16 at 12:45
• Thank you @Xi'an, I will continue to use statistical inference in my paper and presentations. – discretetimeisnice Nov 3 '16 at 10:19

Probabilistic inference uses probabilistic models, i.e. models that describe the statistical problems in terms of probability theory and probability distributions. While statistics use probability theory quite heavily, you cannot say that those two disciplines are the same thing (check the discussion in this thread). Notice that many statistical and machine learning methods do not explicitly use probability theory to define the problems, e.g. many clustering algorithms, or classification methods that work by minimizing some loss function etc. But the distinction is not that straightforward, take as example approximate Bayesian computation -- theoretically it is based on Bayesian (probabilistic!) inference, but it deals with cases where we do not have likelihood function, so instead of it we use a distance measure.

• (+1): I feel less confident about the distinction as I would not classify statistics outside probabilistic modelling. Given that an essential tenet of statistics is to include an assessment of uncertainty, and that this uncertainty need be modelled by a probabilistic structure, I do not see how statistics escapes this framework. – Xi'an Nov 2 '16 at 12:12
• I also object to the ABC link. Although it is nice to see it mentioned, ABC is on the opposite a good illustration of probabilistic modelling as it replicates the exact production of a random sample according to a given probabilistic model! That the likelihood function cannot be numerically computed does not mean that the probability model does not exit, only that it has to be considered from another perspective. – Xi'an Nov 2 '16 at 12:14
• @Xi'an I agree with your comments, but if one wants to make such distinction between probabilistic and non-probabilistic models, then it could be defined as above. Nonetheless, the distinction is quite abstract, ambiguous and in many cases useless, for the reasons you gave. As about ABC, it is just an illustration of case where things get blurry and the distinction becomes ambiguous. – Tim Nov 2 '16 at 12:17
• "classification methods that work by minimizing some loss function", can you provide an example of such a method that couldn't be formulated as a probabilistic model? – nbro Jan 16 '19 at 16:50
• @nbro I said "do not use explicitly...", not that they cannot be formulated like this. – Tim Jan 16 '19 at 17:09

I will answer your questions from my experience in learning Probabilistic Graphical Models (PGM) in university and the way my PGM teacher defined probabilistic inference. Knowing the material of this class was based on [1], I assume you could find more precise answers in this book.

In answer to 2: Probabilistic inference is a type of statistical inference. From [2] and [3], statistical inference makes statistical propositions about a population, which includes point estimate, interval estimate, hypothesis rejection, clustering and classification. "Probabilistic inference" was introduced and roughly defined in the PGM context as any marginalisation task of a probability function, whether it is a marginal probability computation or finding the most probable outcome (for e.g. classification). It thus enters in the definition of statistical inference as making a proposition on the underlying probability distribution of the population.

To illustrate mathematically some marginalisation tasks in the context of PGM, let $$\mathcal{X} = \{X_1, \ldots, X_n\}$$ be a set of random variables. For a given bayesian network $$(\mathscr{G}, \mathbb{P}_{\theta})$$ or markovian network $$(\mathscr{H}, \mathbb{P}_{\theta})$$ with $$\mathbb{P}_\theta$$, then the following routines are considered probabilistic inferences:

• Computing a marginal or conditional probablity: For $$\mathbf{E}, \mathbf{X} \subset \mathcal{X}$$, we want to answer: $$\mathbb{P}_\theta(\mathbf{X} = \mathbf{x} \mid \mathbf{E} = \mathbf{e}) = ~?.$$

• Most probable realization: For $$\mathbf{E}, \mathbf{X} \subset \mathcal{X}$$, we want to answer: $${\rm arg}\min_{\!\!\!\!\!\!\!\!\mathbf{x}}\mathbb{P}_\theta(\mathbf{X} = \mathbf{x} \mid \mathbf{E} = \mathbf{e}) = ~?.$$

In answer to 1 and 3: It was the first time I had seen this terminology. The term does make sense, since you make inference on questions that are directly related to probabilities. I cannot answer whether it is used solely in CS or the PGM context.

[1] Koller, Daphne, and Nir Friedman. 2009. Probabilistic Graphical Models: Principles and Techniques. MIT Press.
[2] https://en.wikipedia.org/wiki/Statistical_inference
[3] https://encyclopediaofmath.org/wiki/Statistical_inference