60
The Dirichlet distribution is a conjugate prior for the multinomial distribution. This means that if the prior distribution of the multinomial parameters is Dirichlet then the posterior distribution is also a Dirichlet distribution (with parameters different from those of the prior). The benefit of this is that (a) the posterior distribution is easy to ...
26
A prior for a parameter will almost always have some specific functional form (written in terms of the density, generally). Let's say we restrict ourselves to one particular family of distributions, in which case choosing our prior reduces to choosing the parameters of that family.
For example, consider a normal model $Y_i \stackrel{_\text{iid}}{\sim} N(\mu,\...
25
It seems that you already gave up on conjugacy. Just for the record, one thing that I've seen people doing (but don't remember exactly where, sorry) is a reparameterization like this. If $X_1,\dots,X_n$ are conditionally iid, given $\alpha,\beta$, such that $X_i\mid\alpha,\beta\sim\mathrm{Beta}(\alpha,\beta)$, remember that
$$
\mathbb{E}[X_i\mid\alpha,\...
25
Yes, it has a conjugate prior in the exponential family. Consider the three parameter family
$$
\pi(\alpha, \beta \mid a, b, p)
\propto \left\{\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}\right\}^p
\exp\left(a\alpha + b\beta \right).
$$
For some values of $(a, b, p)$ this is integrable, although I haven't quite figured out which (I ...
25
The basic idea of Bayesian updating is that given some data $X$ and prior over parameter of interest $\theta$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior
$$ p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta) $$
This can be done sequentially, where after seeing first ...
19
In addition rather than contradiction to Måns T's answer, I simply point out that there is no such thing as "the prior" in Bayesian modelling! The Dirichlet distribution is a convenient choice because of (a) conjugacy, (b) computing, and (c) connection with non-parametric statistics (since this is the discretised version of the Dirichlet process).
However, ...
19
The Oxford English Dictionary defines "conjugate" as an adjective meaning "joined together, esp. in a pair, coupled; connected, related." It's not a huge stretch to imagine that a conjugate prior has a special and strong connection to its posterior.
It's used in a similar sense in chemistry (conjugate acid/base; conjugate solution), ...
16
Maybe satisfying the category "heuristic" justification, conjugate priors are useful because, among others, of the "fictitious sample interpretation".
For example, in the Beta-Bernoulli case, the conjugate prior is Beta with density $$ \pi \left( \theta \right) =\frac{\Gamma \left( \alpha _{0}+\beta _{0}\right)
}{\Gamma \left( \alpha _{0}\right) \Gamma \...
15
At least sort of. Let's look at them one at a time first (taking the other as given).
From the link (with the modification of following the convention of using Greek symbols for parameters):
$f(x|\mu,\tau) = \frac{1}{2\tau} \exp \left( -\frac{|x-\mu|}{\tau} \right) \,$
- scale parameter:
$\cal{L}(\tau) \propto \tau^{-k-1} e^{-\frac{S}{\tau}} \,$
for certain ...
11
It is not by accident. Here you shall find a brief a very nice review on conjugate priors. Concretely, it mentions that if there exist a set of of sufficient statistics of fi xed dimension for the given likelihood function, then you can construct a conjugate prior for it. Having a set of sufficient statistics means that you can factorize the likelihood in a ...
11
As explained for example in Section 3.3.3 of book "The Bayesian choice" by Christian Robert, there is indeed a narrow connection between exponential families and conjugate priors, but there are conjugate priors available for certain non-exponential families. He calls these "quasi-exponential", however, because they are families for which sufficient ...
10
If your model belongs to an exponential family, that is, if the density of the distribution is of the form
$$f(x|\theta)=h(x)\exp\{T(\theta)\cdot S(x)-\psi(\theta)\}\qquad
x\in\mathcal{X}\quad\theta\in\Theta$$
with respect to a given dominating measure (Lebesgue, counting, &tc.), where $t\cdot s$ denotes a scalar product over $\mathbb{R}^d$ and
$$T:\...
10
The point is that we know what the posterior is proportional to and it so happens that we do not need to do the integration to get the (constant) denominator, because we recognise that a distribution with probability density function proportional to $x^{\alpha-1} \times (1-x)^{\beta-1}$ (such as the posterior) is a beta distribution. Since the normalizing ...
9
In theory there should be a conjugate prior for the beta distribution. This is because
the beta distribution is one of the exponential family distributions, and
in theory it should be possible to derive a prior. See, e.g., wikipedia, D Blei's lecture on exponential families.
However the derivation looks difficult, and to quote A Bouchard-Cote's Exponential ...
9
I think it is more correct to speak of the posterior distribution of your parameter $\sigma'^{2}$ rather than its posterior estimate. For clarity of notations, I will drop the prime in $\sigma'^{2}$ in what follows.
Suppose that $X$ is distributed as $\mathcal{N}(0, \sigma^2)$, — I drop $\mu$ for now to make a heuristic example — and $1/\sigma^2 = \sigma^{-...
8
You're looking for the hypergeometric distribution. Incidentally, this link is one of the first two hits on Google for "beta binomial" "conjugate prior".
8
If we take your question to mean whether the product of the densities are Gaussian, then the answer is "yes".
Take $f(x)$ and $g(x)$ to be two normal densities with means $\mu_f$ and $\mu_g$ and variances $\sigma_f^2$ and $\sigma_g^2$.
The producct is $$f(x)g(x)=\frac{1}{2\pi\sigma_f\sigma_g}\exp\left(-\frac{(x-\mu_f)^2}{2\sigma_f^2}+\frac{(x-\mu_g)^2}{2\...
8
Calculating posteriors with general/arbitrary priors directly may be a difficult task.
On the other hand, calculating posteriors with mixtures of conjugate priors is relatively simple, since a given mixture of priors becomes the same mixture of the corresponding posteriors.
[There are also many cases where some given prior may be quite well approximated ...
8
The essential step is along these lines:
$$(x-a)^TA(x-a) + (x-b)^TB(x-b)$$
$$=x^TAx -2a^TAx + a^TAa+ x^TBx -2b^TBx + b^TBb$$
$$=x^T(A+B)x -2(a^TA+b^TB)x + (a^TAa+ b^TBb)$$
$$=x^T(A+B)x -2(a^TA+b^TB)x + (a^TAa+ b^TBb)$$
Let $M=A+B$, let $m=(M^{-1})^T(B^Tb+A^Ta)$ and let $C=a^TAa+ b^TBb-m^TMm$
(I hope I got $m$ right there. You just need to pick $m$ so ...
8
The setup
You have this model:
\begin{align*}
p & \, \sim \, \text{beta}(\alpha, \beta) \\
x \, | \, p & \, \sim \, \text{binomial}(n, p)
\end{align*}
The densities for which are
\begin{equation*}
f(p) = \frac{1}{B(\alpha, \beta)} p^{\alpha - 1} (1 - p)^{\beta - 1}
\end{equation*}
\begin{equation*}
g(x \, | \, p) = {n \choose x} p^x (1 - p)^{n - x}
\...
8
General Remarks
To make the answer given by @Björn a bit more explicit and in the same time more general, we should remember that we arrived at the Bayes Theorem from
$p(\theta|X) \times p(X) = p(X,\theta)=p(X|\theta)\times p(\theta)$
$\implies p(\theta|X) = \frac{p(X|\theta)\times p(\theta)}{p(X)}$ (Bayes Thereom)
where $X$ represents the observed data ...
8
Since
$$L(y_1,\ldots,y_n|\rho)\propto(1-\rho^2)^{-\frac{n}{2}}\exp\bigg\{-\dfrac{\sum_{i=1}^{n}\tilde{y}_{i1}^2 - 2\rho\tilde{y}_{i1}\tilde{y}_{i2}+\tilde{y}_{i2}^2}{2(1-\rho^2)}\bigg \}$$
is a function of $\rho$ of the form
$$(1-\rho^2)^{-\alpha}\exp\bigg\{-\dfrac{\beta}{1-\rho^2}-\dfrac{\gamma\rho}{1-\rho^2}\bigg \}\qquad (1)$$this leads to an exponential ...
8
Quoting verbatim from my book
It is always possible to reduce an exponential family to a standard
and minimal form of dimension $m$,
and this dimension $m$ does not depend on the chosen parameterisation
(Brown, 1986, pp. 13-16). (See Exercise
3.20 for an example of a non-regular exponential family.)
Natural exponential families can also be ...
7
This question is actually somewhat subtle, and it brings to attention an interesting quirk of usage that I hadn't noticed before.
For every practical definition of conjugate distributions that I'm familiar with, it is the case that the posterior of a model using a conjugate prior is a modified form of the prior. The wikipedia definition follows the "...
7
The problem with the Wikipedia article and the reference therein (Fink D., 1997) is that there is some key information missing.
Specifically, the given posterior is for $M-x$ (i.e. the number of target individuals in the population shifted by the number observed in the sample), not for $M$. Furthermore, the posterior parameter corresponding to the number of ...
7
The exponents in the prior density and the likelihood are added to each other
$$
\frac{(\mu-\mu_0)^2}{\tau^2} + \frac{(\overline x - \mu)^2}{\sigma^2/n}
\quad = \quad \frac{\sigma^2(\mu-\mu_0)^2 + \tau^2(\overline x - \mu)^2}{\sigma^2\tau^2/n} \tag 1
$$
Now let's work on the numerator:
$$
((\sigma^2/n)+\tau^2) \Big(\mu^2 - 2(\mu_0\sigma^2 + \overline x \tau^...
6
It is an interesting remark that I have not seen explicitly spelled out, however the parameter space for conjugate priors is often chosen in the opposite way, namely the largest possible set that keeps the sampling distribution well defined. See Brown's Fundamentals of Statistical Exponential Families (1986).
6
Ok, thanks to @Xi'an answer I could make the whole derivation. I will write it for a general case:
\begin{align}
\mathcal{W}(\mathbf{W} | \upsilon, \mathbf{S^{-1}} ) \times \mathcal{W}(\mathbf{S} | \upsilon_0, \mathbf{S_0})
\end{align}
where the $\mathbf{S^{-1}}$ is the key to conjugacy. If we want to use $\mathbf{S}$ then it should be :
\begin{align}
\...
6
I do not think this has anything to do with a wrong definition of the Dirichlet prior or posterior: simply, when
$$(x_1,\ldots,x_k)\sim\mathcal{D}(\alpha_1,...,\alpha_k)$$
the mean is given by
$$\mathbb{E}[(x_1,\ldots,x_k)]=\dfrac{(\alpha_1,...,\alpha_k)}{\sum_{i=1}^k\alpha_i}$$
which explains for the discrepancy with the MLE.
>c(11.,4.,5.)/sum(c(11.,4.,...
6
As you said, conjugate priors make things easier and an additional nice property is that when you refresh the model using the posterior as the new prior things are nice and consistent. For instance you can refresh a specific analysis with new data every month, substitute the prior with the old posterior see how the parameters of the posterior distribution ...
Only top voted, non community-wiki answers of a minimum length are eligible
