57
The Dirichlet distribution is a conjugate prior for the multinomial distribution. This means that if the prior distribution of the multinomial parameters is Dirichlet then the posterior distribution is also a Dirichlet distribution (with parameters different from those of the prior). The benefit of this is that (a) the posterior distribution is easy to ...
25
It seems that you already gave up on conjugacy. Just for the record, one thing that I've seen people doing (but don't remember exactly where, sorry) is a reparameterization like this. If $X_1,\dots,X_n$ are conditionally iid, given $\alpha,\beta$, such that $X_i\mid\alpha,\beta\sim\mathrm{Beta}(\alpha,\beta)$, remember that
$$
\mathbb{E}[X_i\mid\alpha,\...
23
Yes, it has a conjugate prior in the exponential family. Consider the three parameter family
$$
\pi(\alpha, \beta \mid a, b, p)
\propto \left\{\frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}\right\}^p
\exp\left(a\alpha + b\beta \right).
$$
For some values of $(a, b, p)$ this is integrable, although I haven't quite figured out which (I ...
18
The basic idea of Bayesian updating is that given some data $X$ and prior over parameter of interest $\theta$, where the relation between data and parameter is described using likelihood function, you use Bayes theorem to obtain posterior
$$ p(\theta \mid X) \propto p(X \mid \theta) \, p(\theta) $$
This can be done sequentially, where after seeing first ...
17
In addition rather than contradiction to Måns T's answer, I simply point out that there is no such thing as "the prior" in Bayesian modelling! The Dirichlet distribution is a convenient choice because of (a) conjugacy, (b) computing, and (c) connection with non-parametric statistics (since this is the discretised version of the Dirichlet process).
However, ...
16
Maybe satisfying the category "heuristic" justification, conjugate priors are useful because, among others, of the "fictitious sample interpretation".
For example, in the Beta-Bernoulli case, the conjugate prior is Beta with density $$ \pi \left( \theta \right) =\frac{\Gamma \left( \alpha _{0}+\beta _{0}\right)
}{\Gamma \left( \alpha _{0}\right) \Gamma \...
14
A prior for a parameter will almost always have some specific functional form (written in terms of the density, generally). Let's say we restrict ourselves to one particular family of distributions, in which case choosing our prior reduces to choosing the parameters of that family.
For example, consider a normal model $Y_i \stackrel{_\text{iid}}{\sim} N(\mu,...
12
Let's look at them one at a time first (taking the other as given).
From the link (with the modification of following the convention of using Greek symbols for parameters):
$f(x|\mu,\tau) = \frac{1}{2\tau} \exp \left( -\frac{|x-\mu|}{\tau} \right) \,$
- scale parameter:
$\cal{L}(\tau) \propto \tau^{-k-1} e^{-\frac{S}{\tau}} \,$
for certain values of $k$...
10
The point is that we know what the posterior is proportional to and it so happens that we do not need to do the integration to get the (constant) denominator, because we recognise that a distribution with probability density function proportional to $x^{\alpha-1} \times (1-x)^{\beta-1}$ (such as the posterior) is a beta distribution. Since the normalizing ...
9
I think it is more correct to speak of the posterior distribution of your parameter $\sigma'^{2}$ rather than its posterior estimate. For clarity of notations, I will drop the prime in $\sigma'^{2}$ in what follows.
Suppose that $X$ is distributed as $\mathcal{N}(0, \sigma^2)$, — I drop $\mu$ for now to make a heuristic example — and $1/\sigma^2 = \sigma^{-...
9
In theory there should be a conjugate prior for the beta distribution. This is because
the beta distribution is one of the exponential family distributions, and
in theory it should be possible to derive a prior. See, e.g., wikipedia, D Blei's lecture on exponential families.
However the derivation looks difficult, and to quote A Bouchard-Cote's Exponential ...
9
As explained for example in Section 3.3.3 of book "The Bayesian choice" by Christian Robert, there is indeed a narrow connection between exponential families and conjugate priors, but there are conjugate priors available for certain non-exponential families. He calls these "quasi-exponential", however, because they are families for which sufficient ...
8
You're looking for the hypergeometric distribution. Incidentally, this link is one of the first two hits on Google for "beta binomial" "conjugate prior".
8
If we take your question to mean whether the product of the densities are Gaussian, then the answer is "yes".
Take $f(x)$ and $g(x)$ to be two normal densities with means $\mu_f$ and $\mu_g$ and variances $\sigma_f^2$ and $\sigma_g^2$.
The producct is $$f(x)g(x)=\frac{1}{2\pi\sigma_f\sigma_g}\exp\left(-\frac{(x-\mu_f)^2}{2\sigma_f^2}+\frac{(x-\mu_g)^2}{2\...
8
Calculating posteriors with general/arbitrary priors directly may be a difficult task.
On the other hand, calculating posteriors with mixtures of conjugate priors is relatively simple, since a given mixture of priors becomes the same mixture of the corresponding posteriors.
[There are also many cases where some given prior may be quite well approximated ...
8
The setup
You have this model:
\begin{align*}
p & \, \sim \, \text{beta}(\alpha, \beta) \\
x \, | \, p & \, \sim \, \text{binomial}(n, p)
\end{align*}
The densities for which are
\begin{equation*}
f(p) = \frac{1}{B(\alpha, \beta)} p^{\alpha - 1} (1 - p)^{\beta - 1}
\end{equation*}
\begin{equation*}
g(x \, | \, p) = {n \choose x} p^x (1 - p)^{n - x}
\...
8
Since
$$L(y_1,\ldots,y_n|\rho)\propto(1-\rho^2)^{-\frac{n}{2}}\exp\bigg\{-\dfrac{\sum_{i=1}^{n}\tilde{y}_{i1}^2 - 2\rho\tilde{y}_{i1}\tilde{y}_{i2}+\tilde{y}_{i2}^2}{2(1-\rho^2)}\bigg \}$$
is a function of $\rho$ of the form
$$(1-\rho^2)^{-\alpha}\exp\bigg\{-\dfrac{\beta}{1-\rho^2}-\dfrac{\gamma\rho}{1-\rho^2}\bigg \}\qquad (1)$$this leads to an exponential ...
8
Quoting verbatim from my book
It is always possible to reduce an exponential family to a standard
and minimal form of dimension $m$,
and this dimension $m$ does not depend on the chosen parameterisation
(Brown, 1986, pp. 13-16). (See Exercise
3.20 for an example of a non-regular exponential family.)
Natural exponential families can also be ...
7
It is not by accident. Here you shall find a brief a very nice review on conjugate priors. Concretely, it mentions that if there exist a set of of sufficient statistics of fi xed dimension for the given likelihood function, then you can construct a conjugate prior for it. Having a set of sufficient statistics means that you can factorize the likelihood in a ...
7
General Remarks
To make the answer given by @Björn a bit more explicit and in the same time more general, we should remember that we arrived at the Bayes Theorem from
$p(\theta|X) \times p(X) = p(X,\theta)=p(X|\theta)\times p(\theta)$
$\implies p(\theta|X) = \frac{p(X|\theta)\times p(\theta)}{p(X)}$ (Bayes Thereom)
where $X$ represents the observed data ...
7
This question is actually somewhat subtle, and it brings to attention an interesting quirk of usage that I hadn't noticed before.
For every practical definition of conjugate distributions that I'm familiar with, it is the case that the posterior of a model using a conjugate prior is a modified form of the prior. The wikipedia definition follows the "...
6
It is an interesting remark that I have not seen explicitly spelled out, however the parameter space for conjugate priors is often chosen in the opposite way, namely the largest possible set that keeps the sampling distribution well defined. See Brown's Fundamentals of Statistical Exponential Families (1986).
6
Ok, thanks to @Xi'an answer I could make the whole derivation. I will write it for a general case:
\begin{align}
\mathcal{W}(\mathbf{W} | \upsilon, \mathbf{S^{-1}} ) \times \mathcal{W}(\mathbf{S} | \upsilon_0, \mathbf{S_0})
\end{align}
where the $\mathbf{S^{-1}}$ is the key to conjugacy. If we want to use $\mathbf{S}$ then it should be :
\begin{align}
\...
6
The essential step is along these lines:
$$(x-a)^TA(x-a) + (x-b)^TB(x-b)$$
$$=x^TAx -2a^TAx + a^TAa+ x^TBx -2b^TBx + b^TBb$$
$$=x^T(A+B)x -2(a^TA+b^TB)x + (a^TAa+ b^TBb)$$
$$=x^T(A+B)x -2(a^TA+b^TB)x + (a^TAa+ b^TBb)$$
Let $M=A+B$, let $m=(M^{-1})^T(B^Tb+A^Ta)$ and let $C=a^TAa+ b^TBb-m^TMm$
(I hope I got $m$ right there. You just need to pick $m$ so ...
6
I do not think this has anything to do with a wrong definition of the Dirichlet prior or posterior: simply, when
$$(x_1,\ldots,x_k)\sim\mathcal{D}(\alpha_1,...,\alpha_k)$$
the mean is given by
$$\mathbb{E}[(x_1,\ldots,x_k)]=\dfrac{(\alpha_1,...,\alpha_k)}{\sum_{i=1}^k\alpha_i}$$
which explains for the discrepancy with the MLE.
>c(11.,4.,5.)/sum(c(11.,4.,...
6
The exponents in the prior density and the likelihood are added to each other
$$
\frac{(\mu-\mu_0)^2}{\tau^2} + \frac{(\overline x - \mu)^2}{\sigma^2/n}
\quad = \quad \frac{\sigma^2(\mu-\mu_0)^2 + \tau^2(\overline x - \mu)^2}{\sigma^2\tau^2/n} \tag 1
$$
Now let's work on the numerator:
$$
((\sigma^2/n)+\tau^2) \Big(\mu^2 - 2(\mu_0\sigma^2 + \overline x \tau^...
6
A generic if rarely mentioned result about conjugate families is that they are defined in terms of an arbitrary dominating measure $\lambda$. This means that their density wrt this dominating measure is provided by the corresponding exponential family shape
$$\exp\{A(\theta)\cdot S_0 -\lambda \psi(\theta)\}$$
but that the dominating measure $\lambda$ may ...
5
There's no conjugate prior for this likelihood. Likelihoods that admit conjugate distributions correspond to data distributions that are members of some exponential family. Having a non-linear function of the parameters in the log-likelihood makes it impossible for the data distribution to belong to an exponential family.
Even though there's no conjugate ...
5
This seems to be a fairly common issue when beginning to work with these problems; I've had similar questions quite a few times in person.
As you suggest in comments, the normalizing constant is implied by the fact that you must be left with a density at the end.
Slightly less informally, at every step of the derivation, we're really dealing with something ...
5
To go from the third to fourth row just ignore factors that are constant with respect to $\pi$. That is, let
$$\binom{n}{y}\frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}=k$$
so
$$p(\pi|y)=k \cdot \pi^y (1-\pi)^{(n-y)}\pi^{(\alpha-1)}(1-\pi)^{(\beta-1)}$$
or
$$p(\pi|y) \propto \pi^y(1-\pi)^{(n-y)}\pi^{(\alpha-1)}(1-\pi)^{(\beta-1)}$$
The idea'...
Only top voted, non community-wiki answers of a minimum length are eligible
