I would like to clarify a basic question about how to use the concepts of marginal, conditional, and likelihood distributions.

When we observe data from phenomena $x$. We are looking at realizations from the marginal distribution $f(x)$?

When we want to say something about a parameter $\theta$ governing that phenomena, we are usually interested in its posterior distribution $f(\theta| x)$?

And in order to investigate that conditional distribution we make use of prior information and the hypothesized relationship between $\theta$ and the data $x$. We call this later relationship the likelihood distribution $f(x|\theta)$?

Is this the write terminology? If not I would appreciate clarifications where needed. Thanks


Let's start by talking about a marginal distribution. Whenever we have a joint probability distribution $P(X, Y)$, if we draw samples from this distribution and only look at $X$ we have essentially samples from the marginal distribution $P(X)$. The marginal distribution removes $Y$ from the picture by summing or integrating $P(X,Y)$ over all possible values that $Y$ can take.

$$ P(X=x) = \int P(X=x, Y=y) dy $$

In your examples you were mainly dealing with Bayesian Perimetric Models. Let's consider the following model:

$$ \mu \sim \mathcal{N}(0, 1), $$

$$ X_i \sim \mathcal{N}(\mu, 1). $$

Let's refer to $(X_1, \dots, X_n)$ as $X$. In this problem we have observed $X$ and we want to understand $\mu$. With our model, if we knew $\mu$ calculating the probability of each $X_i$ would be easy. So we can easily evaluate the conditional probability $P(X_i | \mu)$. Because of the importance of this distribution in maximum likelihood estimation this is referred to as the likelihood distribution.

As you said in your question we are ultimately interested in $P(\mu | X)$. This is another conditional distribution that we refer to as the posterior distribution ("post" or after we've seen data). Bayes' Theorem tells us how to relate the Likelihood distribution and the Posterior Distribution. However, in this formulation two more factors appear including the (frequentists call it subjective and this is a marginal but don't think about it too hard) prior distribution $P(\mu)$ and the (often horrible to calculate) marginal distribution $P(X)$.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.