How does the marginal distribution become the prior distribution?

Question

Bayes' Theorem for densities/pmf's states that, given, say, two univariate random variables $x,z$ we have

$$p(z\mid x) = \frac {p(x\mid z)\cdot p(z)}{p(x)}$$

This is part of the core of the probability theory most widely used nowadays.

But in order to use this theoretical result for Bayesian statistical inference, two basic mappings are apparently made:

• First, $p(x\mid z)$, goes from being a conditional density to "the likelihood of the sample", a step also used in the maximum likelihood method.

• Second, $p(z)$, goes from being the marginal density to the "prior" density of $z$.

I am interested in this second mapping: can somebody lay out or suggest some literature where this step of going from a marginal density to a "prior" density is discussed in either technical or philosophical detail?

I’m guessing any examples from the literature would be rather "old,” because after decades of Bayesian statistics, calling $p(z)$ the "prior density" has become self-evident.

But to write "$p(z)$ is the prior density that incorporates all the prior beliefs/knowledge about $z$ except the sample at hand" raises the question: "where is the marginal density of a random variable defined in such a way? And if it is not defined in such a way (and it is not), then, what are the arguments that assert that the marginal density is indeed the proper object to represent what we want the "prior density" to represent?"

Answer 1

The answer is that, to set up a problem that makes statistical sense, z and x are not the same kinds of quantities, despite the notational similarity.

• z is parameter that cannot be directly observed.
• x is observable data.

With those constraints in place, the likelihood piece is truly trivial, i.e. P(x|z), the probability of a set of observations given a parameter, is the definition of likelihood.

Statisticians have been arguing about the second part for centuries, but without using the "prior" terminology, P(z) is the probability that the parameter z has a certain value, as has been determined in the absence of the set of observations x. One condition where x definitely has not be incorporated into the determination of z is before the observations x are known, so the term "prior" is appropriate.

Answer 2

...what are the arguments that assert that the marginal density is indeed the proper object to represent what we want the "prior density" to represent?"

At an essential level, I think the argument is that this distribution is "prior" to the data because it does not condition on the data ---i.e., the marginal distribution of the parameters is inherently "prior" to the data because if it were not, it would need to condition on the data. This seems to me to be the essence of the prior/posterior difference.