33 votes
Accepted

Bayesian updating with new data

The basic idea of Bayesian updating is that given some data $X$ and prior over parameter of interest $\theta$, where the relation between data and parameter is described using likelihood function, you ...
user avatar
  • 115k
29 votes

Can anyone explain conjugate priors in simplest possible terms?

A prior for a parameter will almost always have some specific functional form (written in terms of the density, generally). Let's say we restrict ourselves to one particular family of distributions, ...
user avatar
  • 261k
19 votes
Accepted

What is the origin of the name "conjugate prior"?

The Oxford English Dictionary defines "conjugate" as an adjective meaning "joined together, esp. in a pair, coupled; connected, related." It's not a huge stretch to imagine that a ...
user avatar
  • 19.4k
17 votes
Accepted

Justification for conjugate prior?

Maybe satisfying the category "heuristic" justification, conjugate priors are useful because, among others, of the "fictitious sample interpretation". For example, in the Beta-Bernoulli case, the ...
user avatar
17 votes
Accepted

Aside from the exponential family, where else can conjugate priors come from?

As explained for example in Section 3.3.3 of book "The Bayesian choice" by Christian Robert, there is indeed a narrow connection between exponential families and conjugate priors, but there are ...
user avatar
14 votes

beta-binomial as conjugate to hypergeometric

The problem with the Wikipedia article and the reference therein (Fink D., 1997) is that there is some key information missing. Specifically, the given posterior is for $M-x$ (i.e. the number of ...
user avatar
  • 243
13 votes

Gaussian is conjugate of Gaussian?

If we take your question to mean whether the product of the densities are Gaussian, then the answer is "yes" (P.A. Bromiley. Tina Memo No. 2003-003. "Products and Convolutions of ...
user avatar
  • 79.4k
12 votes

Can anyone explain conjugate priors in simplest possible terms?

If your model belongs to an exponential family, that is, if the density of the distribution is of the form $$f(x|\theta)=h(x)\exp\{T(\theta)\cdot S(x)-\psi(\theta)\}\qquad x\in\mathcal{X}\quad\theta\...
user avatar
  • 92.6k
11 votes
Accepted

completing the square for Gaussian multivariate estimation

The essential step is along these lines: $$(x-a)^TA(x-a) + (x-b)^TB(x-b)$$ $$=x^TAx -2a^TAx + a^TAa+ x^TBx -2b^TBx + b^TBb$$ $$=x^T(A+B)x -2(a^TA+b^TB)x + (a^TAa+ b^TBb)$$ $$=x^T(A+B)x -2(a^TA+b^...
user avatar
  • 261k
10 votes
Accepted

Understanding the Beta conjugate prior in Bayesian inference about a frequency

The point is that we know what the posterior is proportional to and it so happens that we do not need to do the integration to get the (constant) denominator, because we recognise that a distribution ...
user avatar
  • 22.9k
8 votes
Accepted

Why is the mixtures of conjugate priors important?

Calculating posteriors with general/arbitrary priors directly may be a difficult task. On the other hand, calculating posteriors with mixtures of conjugate priors is relatively simple, since a given ...
user avatar
  • 261k
8 votes

What are the parameters of a Wishart-Wishart posterior?

Ok, thanks to @Xi'an answer I could make the whole derivation. I will write it for a general case: \begin{align} \mathcal{W}(\mathbf{W} | \upsilon, \mathbf{S^{-1}} ) \times \mathcal{W}(\mathbf{S} | \...
user avatar
  • 2,686
8 votes

Understanding the Beta conjugate prior in Bayesian inference about a frequency

The setup You have this model: \begin{align*} p & \, \sim \, \text{beta}(\alpha, \beta) \\ x \, | \, p & \, \sim \, \text{binomial}(n, p) \end{align*} The densities for which are \begin{...
user avatar
  • 1,426
8 votes

Understanding the Beta conjugate prior in Bayesian inference about a frequency

General Remarks To make the answer given by @Björn a bit more explicit and in the same time more general, we should remember that we arrived at the Bayes Theorem from $p(\theta|X) \times p(X) = p(X,\...
user avatar
  • 256
8 votes
Accepted

Bayesian inference on the correlation parameter of a bivariate normal

Since $$L(y_1,\ldots,y_n|\rho)\propto(1-\rho^2)^{-\frac{n}{2}}\exp\bigg\{-\dfrac{\sum_{i=1}^{n}\tilde{y}_{i1}^2 - 2\rho\tilde{y}_{i1}\tilde{y}_{i2}+\tilde{y}_{i2}^2}{2(1-\rho^2)}\bigg \}$$ is a ...
user avatar
  • 92.6k
8 votes
Accepted

Haldane's prior Beta(0,0) - Part 1

Haldane prior is beta distribution with parameters $\alpha = \beta = 0$. So it is $$ f(p) = \frac{p^{\alpha-1} (1-p)^{\beta-1}}{B(\alpha, \beta)} = \frac{p^{-1}(1-p)^{-1}}{B(0, 0)} $$ where $B(0, 0)$...
user avatar
  • 115k
8 votes
Accepted

Some questions about exponential families

Quoting verbatim from my book It is always possible to reduce an exponential family to a standard and minimal form of dimension $m$, and this dimension $m$ does not depend on the chosen ...
user avatar
  • 92.6k
8 votes

Does a sufficient statistic imply the existence of a conjugate prior?

If there exists a finite dimensional conjugate family, $$\mathfrak F=\{\pi(\cdot|\alpha)\,;\ \alpha\in A\}$$ with $\dim(A)=d$, this means that, for any $\alpha\in A$, there exists a mapping $\tilde\...
user avatar
  • 92.6k
7 votes
Accepted

Dirichlet Prior for Multinomial

I do not think this has anything to do with a wrong definition of the Dirichlet prior or posterior: simply, when $$(x_1,\ldots,x_k)\sim\mathcal{D}(\alpha_1,...,\alpha_k)$$ the mean is given by $$\...
user avatar
  • 92.6k
7 votes
Accepted

Do conjugate priors just lead to a posterior that is a modification of the parameters of the prior?

This question is actually somewhat subtle, and it brings to attention an interesting quirk of usage that I hadn't noticed before. For every practical definition of conjugate distributions that I'm ...
user avatar
  • 79.4k
7 votes

Gamma Conjugate Prior & Poisson Process

Your prior is $\lambda\sim\mathcal G(a, b)$, i.e. $$\pi(\lambda)\propto \lambda^{a-1}e^{-b\lambda}.$$ To get the posterior, multiply the prior by the likelihood: $$\begin{eqnarray} \pi\left(\lambda|...
user avatar
  • 1,827
7 votes
Accepted

If both prior and likelihood are Gaussian what can we say about the posterior?

The exponents in the prior density and the likelihood are added to each other $$ \frac{(\mu-\mu_0)^2}{\tau^2} + \frac{(\overline x - \mu)^2}{\sigma^2/n} \quad = \quad \frac{\sigma^2(\mu-\mu_0)^2 + \...
user avatar
7 votes
Accepted

Interpretation of the rate parameter of a Gamma distribution

Although there are certainly other ideas, here is my quite long intuition about the Gamma Distribution and its parameters: When we sum $\alpha$ (integer) independent exponential RVs, with rate ...
user avatar
  • 52.2k
7 votes

Truncated Gamma Distribution

A generic if rarely mentioned result about conjugate families is that they are defined in terms of an arbitrary dominating measure $\lambda$. This means that their density wrt this dominating measure ...
user avatar
  • 92.6k
6 votes
Accepted

What are the parameters of a Wishart-Wishart posterior?

The product of the two densities in $$ p(\boldsymbol{\Lambda_0 | X, \Lambda}, \upsilon, D, \boldsymbol{\Lambda_x}) \propto \mathcal{W}(\boldsymbol{\Lambda} | \upsilon, \boldsymbol{\Lambda_0}) \mathcal{...
user avatar
  • 92.6k
6 votes

Dirichlet conjugate update derivation

There is nothing wrong with this derivation \begin{align} p({\alpha}|{\theta},{\nu},\eta) &\propto p({\alpha},{\theta}|{\nu},\eta)\\ &= f({\theta}|{\alpha})p({\alpha}|{\nu},\eta)\\ &\...
user avatar
  • 92.6k
6 votes
Accepted

Overestimation of the noise precision in Bayesian linear regression when $n\gtrsim p$

This problem turns out to be well-known in the frequentist literature. In particular, if we use an impropr prior $\Lambda_0=b_0=0$, the posterior scale hyperparameter for the distribution on $\tau$ is ...
user avatar
6 votes
Accepted

conjugate prior: is ever the best choice?

As you said, conjugate priors make things easier and an additional nice property is that when you refresh the model using the posterior as the new prior things are nice and consistent. For instance ...
user avatar
6 votes
Accepted

Why do we use inverse Gamma as prior on variance, when empirical variance is Gamma (chi square)

No reconciliation is needed. In one case you are referring to the sampling distribution of the maximum likelihood estimator, which is a function of the data. In the other, you are referring to the ...
user avatar
  • 33.4k
6 votes
Accepted

Why use MCMC sampling when using conjugate priors?

You are correct that if you have a conjugate prior, there's no need to use MCMC as the posterior has a closed form solution. MCMC tutorials that present a problem where we know the posterior already ...
user avatar
  • 18.1k

Only top scored, non community-wiki answers of a minimum length are eligible