33
votes
Accepted
Bayesian updating with new data
The basic idea of Bayesian updating is that given some data $X$ and prior over parameter of interest $\theta$, where the relation between data and parameter is described using likelihood function, you ...
Tim♦
- 115k
29
votes
Can anyone explain conjugate priors in simplest possible terms?
A prior for a parameter will almost always have some specific functional form (written in terms of the density, generally). Let's say we restrict ourselves to one particular family of distributions, ...
- 261k
19
votes
Accepted
What is the origin of the name "conjugate prior"?
The Oxford English Dictionary defines "conjugate" as an adjective meaning "joined together, esp. in a pair, coupled; connected, related." It's not a huge stretch to imagine that a ...
- 19.4k
17
votes
Accepted
Justification for conjugate prior?
Maybe satisfying the category "heuristic" justification, conjugate priors are useful because, among others, of the "fictitious sample interpretation".
For example, in the Beta-Bernoulli case, the ...
- 26.7k
17
votes
Accepted
Aside from the exponential family, where else can conjugate priors come from?
As explained for example in Section 3.3.3 of book "The Bayesian choice" by Christian Robert, there is indeed a narrow connection between exponential families and conjugate priors, but there are ...
- 26.7k
14
votes
beta-binomial as conjugate to hypergeometric
The problem with the Wikipedia article and the reference therein (Fink D., 1997) is that there is some key information missing.
Specifically, the given posterior is for $M-x$ (i.e. the number of ...
- 243
13
votes
Gaussian is conjugate of Gaussian?
If we take your question to mean whether the product of the densities are Gaussian, then the answer is "yes" (P.A. Bromiley. Tina Memo No. 2003-003. "Products and Convolutions of ...
- 79.4k
12
votes
Can anyone explain conjugate priors in simplest possible terms?
If your model belongs to an exponential family, that is, if the density of the distribution is of the form
$$f(x|\theta)=h(x)\exp\{T(\theta)\cdot S(x)-\psi(\theta)\}\qquad
x\in\mathcal{X}\quad\theta\...
- 92.6k
11
votes
Accepted
completing the square for Gaussian multivariate estimation
The essential step is along these lines:
$$(x-a)^TA(x-a) + (x-b)^TB(x-b)$$
$$=x^TAx -2a^TAx + a^TAa+ x^TBx -2b^TBx + b^TBb$$
$$=x^T(A+B)x -2(a^TA+b^TB)x + (a^TAa+ b^TBb)$$
$$=x^T(A+B)x -2(a^TA+b^...
- 261k
10
votes
Accepted
Understanding the Beta conjugate prior in Bayesian inference about a frequency
The point is that we know what the posterior is proportional to and it so happens that we do not need to do the integration to get the (constant) denominator, because we recognise that a distribution ...
- 22.9k
8
votes
Accepted
Why is the mixtures of conjugate priors important?
Calculating posteriors with general/arbitrary priors directly may be a difficult task.
On the other hand, calculating posteriors with mixtures of conjugate priors is relatively simple, since a given ...
- 261k
8
votes
What are the parameters of a Wishart-Wishart posterior?
Ok, thanks to @Xi'an answer I could make the whole derivation. I will write it for a general case:
\begin{align}
\mathcal{W}(\mathbf{W} | \upsilon, \mathbf{S^{-1}} ) \times \mathcal{W}(\mathbf{S} | \...
- 2,686
8
votes
Understanding the Beta conjugate prior in Bayesian inference about a frequency
The setup
You have this model:
\begin{align*}
p & \, \sim \, \text{beta}(\alpha, \beta) \\
x \, | \, p & \, \sim \, \text{binomial}(n, p)
\end{align*}
The densities for which are
\begin{...
- 1,426
8
votes
Understanding the Beta conjugate prior in Bayesian inference about a frequency
General Remarks
To make the answer given by @Björn a bit more explicit and in the same time more general, we should remember that we arrived at the Bayes Theorem from
$p(\theta|X) \times p(X) = p(X,\...
- 256
8
votes
Accepted
Bayesian inference on the correlation parameter of a bivariate normal
Since
$$L(y_1,\ldots,y_n|\rho)\propto(1-\rho^2)^{-\frac{n}{2}}\exp\bigg\{-\dfrac{\sum_{i=1}^{n}\tilde{y}_{i1}^2 - 2\rho\tilde{y}_{i1}\tilde{y}_{i2}+\tilde{y}_{i2}^2}{2(1-\rho^2)}\bigg \}$$
is a ...
- 92.6k
8
votes
Accepted
Haldane's prior Beta(0,0) - Part 1
Haldane prior is beta distribution with parameters $\alpha = \beta = 0$. So it is
$$
f(p) = \frac{p^{\alpha-1} (1-p)^{\beta-1}}{B(\alpha, \beta)} = \frac{p^{-1}(1-p)^{-1}}{B(0, 0)}
$$
where $B(0, 0)$...
Tim♦
- 115k
8
votes
Accepted
Some questions about exponential families
Quoting verbatim from my book
It is always possible to reduce an exponential family to a standard
and minimal form of dimension $m$,
and this dimension $m$ does not depend on the chosen ...
- 92.6k
8
votes
Does a sufficient statistic imply the existence of a conjugate prior?
If there exists a finite dimensional conjugate family,
$$\mathfrak F=\{\pi(\cdot|\alpha)\,;\ \alpha\in A\}$$
with $\dim(A)=d$, this means that, for any $\alpha\in A$, there exists a mapping $\tilde\...
- 92.6k
7
votes
Accepted
Dirichlet Prior for Multinomial
I do not think this has anything to do with a wrong definition of the Dirichlet prior or posterior: simply, when
$$(x_1,\ldots,x_k)\sim\mathcal{D}(\alpha_1,...,\alpha_k)$$
the mean is given by
$$\...
- 92.6k
7
votes
Accepted
Do conjugate priors just lead to a posterior that is a modification of the parameters of the prior?
This question is actually somewhat subtle, and it brings to attention an interesting quirk of usage that I hadn't noticed before.
For every practical definition of conjugate distributions that I'm ...
- 79.4k
7
votes
Gamma Conjugate Prior & Poisson Process
Your prior is $\lambda\sim\mathcal G(a, b)$, i.e.
$$\pi(\lambda)\propto \lambda^{a-1}e^{-b\lambda}.$$
To get the posterior, multiply the prior by the likelihood:
$$\begin{eqnarray}
\pi\left(\lambda|...
- 1,827
7
votes
Accepted
If both prior and likelihood are Gaussian what can we say about the posterior?
The exponents in the prior density and the likelihood are added to each other
$$
\frac{(\mu-\mu_0)^2}{\tau^2} + \frac{(\overline x - \mu)^2}{\sigma^2/n}
\quad = \quad \frac{\sigma^2(\mu-\mu_0)^2 + \...
- 7,244
7
votes
Accepted
Interpretation of the rate parameter of a Gamma distribution
Although there are certainly other ideas, here is my quite long intuition about the Gamma Distribution and its parameters:
When we sum $\alpha$ (integer) independent exponential RVs, with rate ...
- 52.2k
7
votes
Truncated Gamma Distribution
A generic if rarely mentioned result about conjugate families is that they are defined in terms of an arbitrary dominating measure $\lambda$. This means that their density wrt this dominating measure ...
- 92.6k
6
votes
Accepted
What are the parameters of a Wishart-Wishart posterior?
The product of the two densities in
$$
p(\boldsymbol{\Lambda_0 | X, \Lambda}, \upsilon, D, \boldsymbol{\Lambda_x}) \propto
\mathcal{W}(\boldsymbol{\Lambda} | \upsilon, \boldsymbol{\Lambda_0})
\mathcal{...
- 92.6k
6
votes
Dirichlet conjugate update derivation
There is nothing wrong with this derivation
\begin{align}
p({\alpha}|{\theta},{\nu},\eta) &\propto p({\alpha},{\theta}|{\nu},\eta)\\
&= f({\theta}|{\alpha})p({\alpha}|{\nu},\eta)\\
&\...
- 92.6k
6
votes
Accepted
Overestimation of the noise precision in Bayesian linear regression when $n\gtrsim p$
This problem turns out to be well-known in the frequentist literature. In particular, if we use an impropr prior $\Lambda_0=b_0=0$, the posterior scale hyperparameter for the distribution on $\tau$ is
...
- 917
6
votes
Accepted
conjugate prior: is ever the best choice?
As you said, conjugate priors make things easier and an additional nice property is that when you refresh the model using the posterior as the new prior things are nice and consistent. For instance ...
- 1,335
6
votes
Accepted
Why do we use inverse Gamma as prior on variance, when empirical variance is Gamma (chi square)
No reconciliation is needed. In one case you are referring to the sampling distribution of the maximum likelihood estimator, which is a function of the data. In the other, you are referring to the ...
- 33.4k
6
votes
Accepted
Why use MCMC sampling when using conjugate priors?
You are correct that if you have a conjugate prior, there's no need to use MCMC as the posterior has a closed form solution. MCMC tutorials that present a problem where we know the posterior already ...
- 18.1k
Only top scored, non community-wiki answers of a minimum length are eligible