I understand that likelihood differs from a probability distribution because likelihood describes the probability of certain parameter values given the data that you've observed (it's essentially a distribution that describes observed data) while a probability distribution describes the probability of observing certain values given constant parameter values. But what is a marginal likelihood and how does it relate to posterior distributions? (preferably explained without, or with as little as possible, probability notation so that the explanation is more intuitive). Any examples would be great as well.
$\begingroup$
$\endgroup$
4
-
1$\begingroup$ A marginal likelihood just has the effects of other parameters integrated out so that it is a function of just your parameter of interest. For example, suppose your likelihood function takes the form L(x,y,z). The marginal likelihood L(x) is obtained by integrating out the effect of y and z. See (en.wikipedia.org/wiki/Marginal_likelihood) for more detail $\endgroup$ – Forrest Apr 13 at 1:19
-
2$\begingroup$ Does this answer your question? In Bayesian terminology, what does evidence refer to? Also my own answer in another duplicate post $\endgroup$ – user20181201 Apr 13 at 1:20
-
2$\begingroup$ Does this answer your question? What is the role of model likelihood? $\endgroup$ – Arya McCarthy Apr 13 at 1:20
-
2$\begingroup$ See similar question here: stats.stackexchange.com/questions/515801/… $\endgroup$ – ofer-a Apr 13 at 7:21
Add a comment
|
$\begingroup$
$\endgroup$
In Bayesian statistics, the marginal likelihood $$m(x) = \int_\Theta f(x|\theta)\pi(\theta)\,\text d\theta$$ where
- $x$ is the sample
- $f(x|\theta)$ is the sampling density, which is proportional to the model likelihood
- $\pi(\theta)$ is the prior density
is a misnomer in that
- it is not a likelihood function [as a function of the parameter], since the parameter is integrated out (i.e., the likelihood function is averaged against the prior measure),
- it is a density in the observations, the predictive density of the sample,
- it is not defined up to a multiplicative constant,
- it does not solely depend on sufficient statistics
Other names for $m(x)$ are evidence, prior predictive, partition function. It has however several important roles:
- this is the normalising constant of the posterior distribution$$\pi(\theta|x) = \dfrac{f(x|\theta)\pi(\theta)}{m(x)}$$
- in model comparison, this is the contribution of the data to the posterior probability of the associated model and the numerator or denominator in the Bayes factor.
- it is a measure of goodness-of-fit (of a model to the data $x$), in that $2\log m(x)$ is asymptotically the BIC (Bayesian information criterion) of Schwarz (1978).
See also
Normalizing constant in Bayes theorem
$\begingroup$
$\endgroup$
In my mind the most intuitive role of marginal likelihood is indeed as a normalization factor. I'll elaborate more on this.
The bayes theorem is:
$$ P(\phi|X) = \frac{P(X|\phi)P(\phi)} {P(X)} $$
Now let's explore the numerator components:
$$P(X|\phi)P(\phi)$$ Prior multiply by the likelihood using a specific parameter - How well can we explain the data using this specific $\phi$ parameter.
$$P(X)=\int_{\theta}P(X|\theta)P(\theta)\mathrm{d}\theta$$ The marginal likelihood - How well we can explain the data using all the parameters, weighted by the prior.
The ratio between them should give the proportional share of this specific parameter $\phi$. This makes the numerator over the different parameters sum to 1 constructing a probability distribution. Thus the posterior, the ratio between the numerator & denominator, represents the probability distribution over the parameters.
If for example a specific parameter $\phi$ is very good at explaining the data compared to the other weights, then this parameter will have higher probability.
