30
$\begingroup$

I keep seeing the term "inverse probability" mentioned in passing albeit without any explanation.

I know it has to do with Bayesian inference, but what exactly do we mean by inverting a probability?

My thinking at the moment is that in "normal" probability we take outcomes of random experiments and try to estimate probabilities based on the outcomes of these experiments whereas in "inverse probability" we're interested in going from a probability (a prior for an unknown quantity) to knowing the "outcome of an experiment", the experiment being finding out the value of an unknown quantity (i.e. via the posterior, and maybe finding the MAP hypothesis).

That is, in "conventional probability" we go from outcomes of an experiment to probability vs. in inverse probability we go the other way: we go from a prior to uncovering the outcome of an experiment.

Improve this question
$\endgroup$
0

4 Answers 4

Reset to default
25
$\begingroup$

"Inverse probability" is a rather old-fashioned way of referring to Bayesian inference; when it's used nowadays it's usually as a nod to history. De Morgan (1838), An Essay on Probabilities, Ch. 3 "On Inverse Probabilities", explains it nicely:

In the preceding chapter, we have calculated the chances of an event, knowing the circumstances under which it is to happen or fail. We are now to place ourselves in an inverted position: we know the event, and ask what is the probability which results from the event in favour of of any set of circumstances under which the same might have happened.

An example follows using Bayes' Theorem.

I'm not sure that the term mightn't have at some point encompassed putative or proposed non-Bayesian, priorless, methods of getting from $f(y|\theta)$ to $p(\theta|y)$ (in @Christopher Hanck's notation); but at any rate Fisher was clearly distinguishing between "inverse probability" & his methods—maximum likelihood, fiducial inference—by the 1930's. It also strikes me that several early-20th-Century writers seem to view the use of what we now call uninformative/ignorance/reference priors as part & parcel of the "inverse probability" method†, or even of "Bayes' Theorem"‡.

† Fisher (1930), Math. Proc. Camb. Philos. Soc., 26, p 528, "Inverse probability", clearly distinguishes, perhaps for the first time, between Bayesian inference from flat "ignorance" priors ("the inverse argument proper"), the unexceptionable application of Bayes' Theorem when the prior describes aleatory probabilities ("not inverse probability strictly speaking"), & his fiducial argument.

‡ For example, Pearson (1907), Phil. Mag., p365, "On the influence of past experience on future expectation", conflates Bayes' Theorem with the "equal distribution of ignorance".

Improve this answer
$\endgroup$
4
  • 4
    $\begingroup$ +1: the inverse probability problem is about reversing the analytical step from data given parameter to parameter given data, and as far as I can tell includes both Bayesian posterior distributions and simple likelihoods, and possibly more. Fisher certainly seems to have used the phrase in different ways at different times $\endgroup$
    – Henry
    Oct 22, 2020 at 0:00
  • 3
    $\begingroup$ I thought that Fisher's 1930 article 'inverse probability', and his other comments around that time, was meant to be different from the Bayesian method. I would disagree that inverse probability is solely referring to Bayesian posterior probabilities. $\endgroup$
    – Sextus Empiricus
    Oct 22, 2020 at 11:44
  • 2
    $\begingroup$ @SextusEmpiricus: The title doesn't refer to the fiducial probability he's putting forward. Inverse probability is described as "fundamentally false & devoid of foundation" in the first paragraph. But I think I remember other writers using "inverse probability" in a more general sense - you wouldn't be disagreeing with me. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Oct 22, 2020 at 13:40
  • 2
    $\begingroup$ @Scortchi , I agree. Fisher even explicitly mentions how it is false to place 'maximum likelihood' under the same umbrella as 'inverse probability', and the article is a strong critique against inverse probability. But while doing that he also describes the ambiguity and how 'inverse probability' is being perceived as more than strict 'inverse probability'. $\endgroup$
    – Sextus Empiricus
    Oct 22, 2020 at 13:55
22
$\begingroup$

Probability of 'observations' given the 'model'

Typically 'probability' is expressed as the probability of an outcome given a particular experiment/model/setup.

So the probability is about the frequencies of observations given the model. These types of questions are often not so difficult. For instance, in gambling, we can express the probabilities of certain dice rolls or card sequences (and there are many questions here on CV that ask about probability given some situation, which will receive unambiguous and clear answers).

Inverse: inference about the 'model' given the 'observations'

However, in practice, we do not fully know the model, and we wish to infer some unknown properties of the model based on observations. That is, in the inverse direction as probability normally goes. Now the model is unknown, but the observation is given/known. The situation is inverse.

This is a difficult problem. We can express the probabilities of observations given certain models, and we could express the differences in those probabilities for different models, but these expressions are not the same as probabilities for those given models.

Ronald A Fisher's maximum likelihood = inverse probability?

In his 1921 work 'On the mathematical foundations of theoretical statistics' Ronald A. Fisher mentions the method of the maximum likelihood in relation to 'inverse probability'. But he argues that we should not view this 'inverse probability' as a 'probability' and suggests instead the term likelihood.

I must indeed plead guilty in my original statement of the Method of the Maximum Likelihood (9) to having based my argument upon the principle of inverse probability ; in the same paper, it is true, I emphasised the fact that such inverse probabilities were relative only. That is to say, that while we might speak of one value of as having an inverse probability three times that of another value of $p$, we might on no account introduce the differential element $dp$, so as to be able to say that it was three times as probable that $p$ should lie in one rather than the other of two equal elements. Upon consideration, therefore, I perceive that the word probability is wrongly used in such a connection : probability is a ratio of frequencies, and about the frequencies of such values we can know nothing whatever. We must return to the actual fact that one value of $p$, of the frequency of which we know nothing, would yield the observed result three times as frequently as would another value of $p$. If we need a word to characterise this relative property of different values of $p$, I suggest that we may speak without confusion of the likelihood of one value of $p$ being thrice the likelihood of another, bearing always in mind that likelihood is not here used loosely as a synonym of probability, but simply to express the relative frequencies with which such values of the hypothetical quantity $p$ would in fact yield the observed sample.

Inverse probability = Bayesian probability?

Some might say that inverse probability is equal to the posterior Bayesian probability. And it is quite standard as a synonym. But I like to think that it encompasses more than that. All methods of inference are in a way "inverse probability" and try to infer in the opposite direction as the typical probability statement (probability of an outcome given the model).

Yes indeed: only a Bayesian probability is truly/technically a probability. No indeed: a fiducial distribution is not equal to a probability(*). But fiducial and frequentist inference are still just as well about inverting the direction and making statements about parameters given the observation. (The frequentist distributions/intervals are just not technically probabilities.)

(*)The Bayesian posterior distribution is a density of the probability of the parameter conditional on the observation. The fiducial distribution is the density of the confidence and is not relating to the parameter as a random variable, but considers our inference about the random variable as the random factor

Example:

On StackExchange we see two types of questions:

  • I have a fair six-sided die; what is the probability that I roll a 6 six times in a row?
  • I roll a 6 six times in a row with a six-sided die; do I have a fair die?

The first type of question can be answered with a straightforward method and is about expressing the probability of outcomes given a particular situation.

The second type reverses the question. And while the likelihood might be known it does not have the same straightforward answer (it will be a false idea to speak about probability of the die being fair). We could use a Bayesian posterior probability, but still the problem is more general than just applying the Bayesian method.

Wrap up

Inverse probability might relate to Bayesian (posterior) probability, and some might view it in a wider sense (including fiducial "probability" or confidence intervals). But in none of these cases it refers to an actually true probability.

Improve this answer
$\endgroup$
9
  • 2
    $\begingroup$ These concurs with the use in terms like Inverse Problems, see indeed the tag inverse-problem. Bayesian methods is among those used for inverse problems. $\endgroup$
    – kjetil b halvorsen ♦
    Oct 22, 2020 at 14:17
  • 2
    $\begingroup$ (+1) Excellent answer. $\endgroup$
    – Xi'an
    Oct 23, 2020 at 6:06
  • 2
    $\begingroup$ (+1) English is a funny language even for native speakers. By analogy with reverse, reversed, reversing you might expect words like inversed or inversing but they sound odd to me. I've made some small edits correspondingly and naturally hope to preserve your intended meaning. (Die is the singular of dice, although writers with English as their language often don't know that either.) $\endgroup$
    – Nick Cox
    Oct 24, 2020 at 10:35
  • 2
    $\begingroup$ @NickCox I've lost count of how often you have improved my posts, but every time I appreciate it a lot. I never disagree with your improvements. $\endgroup$
    – Sextus Empiricus
    Oct 24, 2020 at 10:38
  • 1
    $\begingroup$ They overlap in meaning: there is a shared sense of moving in the opposite direction (from previous or standard). But in my mathematics reading I don't recall anyone calling $1/x$ the reverse of $x$ or $f^{-1}(x)$ the reverse of $f(x)$, so they certainly aren't synonyms. I think inverse probability inverts the opposite procedure so that could be a good word for you. $\endgroup$
    – Nick Cox
    Oct 24, 2020 at 10:54
6
$\begingroup$

Yes, I believe your thinking is a way to view things in that it points out that the prior is the key ingredient to convert conditional probabilities.

My reading is that it is an interpretation of Bayes' theorem, which, as we know, says $$ P(B|A)=\frac{P(A|B)P(B)}{P(A)}. $$ Hence, Bayes' theorem provides the result to convert one conditional probability into another, whence "inverse" probability.

In the context of statistical applications, $$ p(\theta|y)=\frac{f(y|\theta)p(\theta)}{p(y)}, $$ i.e., we obtain a rule to go from the likelihood $f$ to the posterior $p(\theta|y)$.

Improve this answer
$\endgroup$
0
1
$\begingroup$

There are plenty of great answers already, so I'll add a slightly tangential example that I found intriguing. Hopefully its not too far-off from the topic.

Markov chain Monte Carlo methods are often used for Bayesian posterior inference. In typical encounters of Markov chains in probability theory, we ask questions like whether a chain converges to some stationary distribution in the limit of infinite steps. In Bayesian statistics for example, the inverse question is asked: if we want a chain to converge the posterior distribution of interest, how do we design such a chain? (The standard Metropolis-Hastings (MH) algorithm is one such algorithm.)

This example doesn't directly answering the question, but serves as a fun example of an application of solutions to inverse probability question.

I found this insight from https://www.jstor.org/stable/2684568 which motivates the MH algorithm from the inverse perspective.

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ The convergence of an MCMC algorithm (which is not Bayesian per se but addressing any integration problem) can hardly be considered as "inverse probability". At best, we can see MCMC as inverse engineering: given a target, devise a Markov chain with such stationary distribution. $\endgroup$
    – Xi'an
    Oct 23, 2020 at 5:57
  • $\begingroup$ Ahhh, that's true...!! Per stack exchange rules/etiquette, should I delete my answer because it is factually incorrect? Or should it be kept as an example of an incorrect answer? $\endgroup$
    – user
    Oct 23, 2020 at 6:33
  • 5
    $\begingroup$ There is nothing factually wrong in your answer, as opposed to the next one, it is just mostly unrelated with the question. The reference to Chib and Greenberg (1995) makes a connection of sorts. Hence do not feel obliged to delete it (but then please remove the strike-through style). $\endgroup$
    – Xi'an
    Oct 23, 2020 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.