Probability of 'observations' given the 'model'
Typically 'probability' is expressed as the probability of an outcome given a particular experiment/model/setup.
So the probability is about the frequencies of observations given the model. These types of questions are often not so difficult. For instance, in gambling, we can express the probabilities of certain dice rolls or card sequences (and there are many questions here on CV that ask about probability given some situation, which will receive unambiguous and clear answers).
Inverse: inference about the 'model' given the 'observations'
However, in practice, we do not fully know the model, and we wish to infer some unknown properties of the model based on observations. That is, in the inverse direction as probability normally goes. Now the model is unknown, but the observation is given/known. The situation is inverse.
This is a difficult problem. We can express the probabilities of observations given certain models, and we could express the differences in those probabilities for different models, but these expressions are not the same as probabilities for those given models.
Ronald A Fisher's maximum likelihood = inverse probability?
In his 1921 work 'On the mathematical foundations of theoretical statistics' Ronald A. Fisher mentions the method of the maximum likelihood in relation to 'inverse probability'. But he argues that we should not view this 'inverse probability' as a 'probability' and suggests instead the term likelihood.
I must indeed plead guilty in my original statement of the Method of the Maximum Likelihood (9) to having based my argument upon the principle of inverse probability ; in the same paper, it is true, I emphasised the fact that such inverse probabilities were relative only. That is to say, that while we might speak of one value of as having an inverse probability three times that of another value of $p$, we might on no account introduce the differential element $dp$, so as to be able to say that it was three times as probable that $p$ should lie in one rather than the other of two equal elements. Upon consideration, therefore, I perceive that the word probability is wrongly used in such a connection : probability is a ratio of frequencies, and about the frequencies of such values we can know nothing whatever. We must return to the actual fact that one value of $p$, of the frequency of which we know nothing, would yield the observed result three times as frequently as would another value of $p$. If we need a word to characterise this relative property of different values of $p$, I suggest that we may speak without confusion of the likelihood of one value of $p$ being thrice the likelihood of another, bearing always in mind that likelihood is not here used loosely as a synonym of probability, but simply to express the relative frequencies with which such values of the hypothetical quantity $p$ would in fact yield the observed sample.
Inverse probability = Bayesian probability?
Some might say that inverse probability is equal to the posterior Bayesian probability. And it is quite standard as a synonym. But I like to think that it encompasses more than that. All methods of inference are in a way "inverse probability" and try to infer in the opposite direction as the typical probability statement (probability of an outcome given the model).
Yes indeed: only a Bayesian probability is truly/technically a probability. No indeed: a fiducial distribution is not equal to a probability(*). But fiducial and frequentist inference are still just as well about inverting the direction and making statements about parameters given the observation. (The frequentist distributions/intervals are just not technically probabilities.)
(*)The Bayesian posterior distribution is a density of the probability of the parameter conditional on the observation. The fiducial distribution is the density of the confidence and is not relating to the parameter as a random variable, but considers our inference about the random variable as the random factor
On StackExchange we see two types of questions:
- I have a fair six-sided die; what is the probability that I roll a 6 six times in a row?
- I roll a 6 six times in a row with a six-sided die; do I have a fair die?
The first type of question can be answered with a straightforward method and is about expressing the probability of outcomes given a particular situation.
The second type reverses the question. And while the likelihood might be known it does not have the same straightforward answer (it will be a false idea to speak about probability of the die being fair). We could use a Bayesian posterior probability, but still the problem is more general than just applying the Bayesian method.
Inverse probability might relate to Bayesian (posterior) probability, and some might view it in a wider sense (including fiducial "probability" or confidence intervals). But in none of these cases it refers to an actually true probability.