What is probabilistic inference?

Question

I am reading Chris Bishop's Pattern Recognition and Machine Learning textbook. I came across the term probabilistic inference several times. I have a couple of questions.

1. Is probabilistic inference only applicable in a graphical modelling context?

2. What's the distinction between traditional statistical inference (p-values, confidence intervals, Bayes factors etc.) and probabilistic inference?

3. Is this a term that's specific to the CS community or is it widely used in the statistics community as well?

Answer 1

Probabilistic inference uses probabilistic models, i.e. models that describe the statistical problems in terms of probability theory and probability distributions. While statistics use probability theory quite heavily, you cannot say that those two disciplines are the same thing (check the discussion in this thread). Notice that many statistical and machine learning methods do not explicitly use probability theory to define the problems, e.g. many clustering algorithms, or classification methods that work by minimizing some loss function etc. But the distinction is not that straightforward, take as example approximate Bayesian computation -- theoretically it is based on Bayesian (probabilistic!) inference, but it deals with cases where we do not have likelihood function, so instead of it we use a distance measure.

Answer 2

I will answer your questions from my experience in learning Probabilistic Graphical Models (PGM) in university and the way my PGM teacher defined probabilistic inference. Knowing the material of this class was based on [1], I assume you could find more precise answers in this book.

In answer to 2: Probabilistic inference is a type of statistical inference. From [2] and [3], statistical inference makes statistical propositions about a population, which includes point estimate, interval estimate, hypothesis rejection, clustering and classification. "Probabilistic inference" was introduced and roughly defined in the PGM context as any marginalisation task of a probability function, whether it is a marginal probability computation or finding the most probable outcome (for e.g. classification). It thus enters in the definition of statistical inference as making a proposition on the underlying probability distribution of the population.

To illustrate mathematically some marginalisation tasks in the context of PGM, let $\mathcal{X} = \{X_1, \ldots, X_n\}$ be a set of random variables. For a given bayesian network $(\mathscr{G}, \mathbb{P}_{\theta})$ or markovian network $(\mathscr{H}, \mathbb{P}_{\theta})$ with $\mathbb{P}_\theta$, then the following routines are considered probabilistic inferences:

• Computing a marginal or conditional probablity: For $\mathbf{E}, \mathbf{X} \subset \mathcal{X}$, we want to answer: $$\mathbb{P}_\theta(\mathbf{X} = \mathbf{x} \mid \mathbf{E} = \mathbf{e}) = ~?.$$

• Most probable realization: For $\mathbf{E}, \mathbf{X} \subset \mathcal{X}$, we want to answer: $${\rm arg}\min_{\!\!\!\!\!\!\!\!\mathbf{x}}\mathbb{P}_\theta(\mathbf{X} = \mathbf{x} \mid \mathbf{E} = \mathbf{e}) = ~?.$$

In answer to 1 and 3: It was the first time I had seen this terminology. The term does make sense, since you make inference on questions that are directly related to probabilities. I cannot answer whether it is used solely in CS or the PGM context.

[1] Koller, Daphne, and Nir Friedman. 2009. Probabilistic Graphical Models: Principles and Techniques. MIT Press.
[2] https://en.wikipedia.org/wiki/Statistical_inference
[3] https://encyclopediaofmath.org/wiki/Statistical_inference