Bayes' Theorem for densities/pmf's states that, given, say, two univariate random variables $x,z$ we have

$$p(z\mid x) = \frac {p(x\mid z)\cdot p(z)}{p(x)}$$

This is part of the core of the probability theory most widely used nowadays.

But in order to use this theoretical result for Bayesian statistical inference, two basic mappings are apparently made:

  • First, $p(x\mid z)$, goes from being a conditional density to "the likelihood of the sample", a step also used in the maximum likelihood method.

  • Second, $p(z)$, goes from being the marginal density to the "prior" density of $z$.

I am interested in this second mapping: can somebody lay out or suggest some literature where this step of going from a marginal density to a "prior" density is discussed in either technical or philosophical detail?

I’m guessing any examples from the literature would be rather "old,” because after decades of Bayesian statistics, calling $p(z)$ the "prior density" has become self-evident.

But to write "$p(z)$ is the prior density that incorporates all the prior beliefs/knowledge about $z$ except the sample at hand" raises the question: "where is the marginal density of a random variable defined in such a way? And if it is not defined in such a way (and it is not), then, what are the arguments that assert that the marginal density is indeed the proper object to represent what we want the "prior density" to represent?"


The answer is that, to set up a problem that makes statistical sense, z and x are not the same kinds of quantities, despite the notational similarity.

  • z is parameter that cannot be directly observed.
  • x is observable data.

With those constraints in place, the likelihood piece is truly trivial, i.e. P(x|z), the probability of a set of observations given a parameter, is the definition of likelihood.

Statisticians have been arguing about the second part for centuries, but without using the "prior" terminology, P(z) is the probability that the parameter z has a certain value, as has been determined in the absence of the set of observations x. One condition where x definitely has not be incorporated into the determination of z is before the observations x are known, so the term "prior" is appropriate.

  • 1
    $\begingroup$ Thanks for your input. The problem is that in the definition of the "marginal distribution", there is no special mention of some particular piece of information missing. Rather all "particular" information is missing. On the other hand, the "prior" focuses on the probability if the specific sample is missing from the information set. Essentially the prior is just another conditional probability distribution, not the marginal distribution... maybe Renyi's axiomatization that takes the conditional probability as the fundamental concept is what is needed here.. $\endgroup$ – Alecos Papadopoulos Oct 16 '18 at 17:50
  • $\begingroup$ When you say "cannot be directly observed" how do you deal with different people "observing" different things? Also, the posterior $p(x|z)$ (ie infer $x$ given observed $z$) is just as valid mathematically as $p(z|x)$ is it not? $\endgroup$ – probabilityislogic Jun 27 '20 at 4:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.