On the wikipedia article of Laplace smoothing (or additive smoothing), it is said that from a Bayesian point of view,

this corresponds to the expected value of the posterior distribution, using a symmetric Dirichlet distribution with parameter $\alpha$ as a prior.

I'm puzzled about how that is actually true. Could someone help me understand how those two things are equivalent?



Sure. This is essentially the observation that the Dirichlet distribution is a conjugate prior for the multinomial distribution. This means they have the same functional form. The article mentions it, but I'll just emphasize that this follows from the multinomial sampling model. So, getting down to it...

The observation is about the posterior, so let's introduce some data, $x$, which are counts of $K$ distinct items. We observe $N = \sum_{i=1}^K x_i$ samples total. We'll assume $x$ is drawn from an unknown distribution $\pi$ (on which we'll put a $\mathrm{Dir}(\alpha)$ prior on the $K$-simplex).

The posterior probability of $\pi$ given $\alpha$ and data $x$ is

$$p(\pi | x, \alpha) = p(x | \pi) p(\pi|\alpha)$$

The likelihood, $p(x|\pi)$, is the multinomial distribution. Now let's write out the pdf's:

$$p(x|\pi) = \frac{N!}{x_1!\cdots x_k!} \pi_1^{x_1} \cdots \pi_k^{x_k}$$


$$p(\pi|\alpha) = \frac{1}{\mathrm{B}(\alpha)} \prod_{i=1}^K \pi_i^{\alpha - 1}$$

where $\mathrm{B}(\alpha) = \frac{\Gamma(\alpha)^K}{\Gamma(K\alpha)}$. Multiplying, we find that,

$$ p(\pi|\alpha,x) = p(x | \pi) p(\pi|\alpha) \propto \prod_{i=1}^K \pi_i^{x_i + \alpha - 1}.$$

In other words, the posterior is also Dirichlet. The question was about the posterior mean. Since the posterior is Dirichlet, we can apply the formula for the mean of a Dirichlet to find that,

$$E[\pi_i | \alpha, x] = \frac{x_i + \alpha}{N + K\alpha}.$$

Hope this helps!

  • $\begingroup$ $p(\pi | \alpha, x) = p(x | \pi)p(\pi | \alpha)/p(x | \alpha),$ so isn't it wrong to say that $p(\pi | \alpha, x) = p(x | \pi)p(\pi | \alpha)?$ They are proportional with respect to $\pi$, but writing an equality is not true I think. $\endgroup$ – michal May 18 '16 at 1:18
  • $\begingroup$ I was confused about this for a long time, and I want to share my realization. These folks motivating Laplace smoothing by Dirichlet are using the Posterior Mean, not the MAP. For simplicity, assume the Beta distribution (simplest case of Dirichlet) The posterior mean is $\frac{\alpha + n_{success}}{\alpha + \beta + n_{success} + n_{failures}}$ whereas the MAP is $\frac{\alpha + n_{success} - 1}{\alpha + \beta + n_{success} + n_{failures} - 2}$. So if someone says $\alpha = \beta = 1$ corresponds to adding 1 to numerator and 2 to denominator, it's because they are using the Posterior Mean. $\endgroup$ – RMurphy Jun 19 '17 at 0:26

As a side note, I would also like to add another point to the above derivation, which it's not really concerning the main question. However, talking about Dirichlet priors on multinomial distribution, I thought it worth to mention that what would be the form of likelihood function if we're going to take probabilities as nuisance variables.

As it's correctly pointed out by by sydeulissie, the $p(\pi | \alpha, x)$ is proportional to $\prod_{i=1}^{K} \, \pi_i^{x_i+\alpha-1}$ . Now here I would like to calculate $p(x|\alpha)$.

\begin{equation} p(x | \alpha) = \int \prod_{i=1}^{K}p(x | \pi_i, \alpha)p(\pi|\alpha) \mathrm{d} \pi_1 \mathrm{d} \pi_2 ...\mathrm{d} \pi_K \end{equation}

Using an integral identity for gamma functions, we have: \begin{equation} p(x|\alpha) = \frac{\Gamma(K\alpha)}{\Gamma(N + K\alpha)} \prod_{i=1}^{K} \frac{\Gamma(x_i + \alpha)}{\Gamma(\alpha)} \end{equation}

The above derivation of the likelihood for categorical data proposes a more robust way of dealing with this data for cases that the sample size $N$ is not so big enough.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.