Rate of converge of KL-divergence to posterior Suppose you have samples from some distribution P.  You have a prior distribution Q, which represents your estimate of P, and assume for now that it's parameterized the same way as P.  Upon observing samples from P you perform a Bayesian update of Q, yielding a posterior Q'.
As more samples from P are observed, eventually the Kullback–Leibler divergence between Q' and P gets arbitrarily small, i.e. the KL(Q'||P) -> 0.  My question is: can we write down a rate of converge as a function of the number of samples observed?
This is really a duplication of this question asked by someone else, but it has no satisfactory answers so I thought I'd ask here.
EDIT: I'm particularly interested in the case where P and Q are multinomial distributions.
 A: $$
\newcommand{\KL}{\text{KL}}
\newcommand{\N}{\mathcal{N}}
\newcommand{\norm}[1]{\left|\left|#1\right|\right|}
\newcommand{\Expect}[2]{\mathbb{E}_{#1}\left[#2\right]}
$$
The original question on Math.SE is not well posed, as the KL between the empirical distribution and the true distribution has a non-zero probability of not being finite. You can do something with some assumptions, like trying to estimate some parameters of a functional form.

Assume you want are estimating the mean $\mu$ of a normal distribution with known variance $\sigma^2$ using the sample mean $\bar{x}$.
In this case, the KL divergence reduces to 
$$
\KL(\N(\bar{x}, \sigma^2) || \N(\mu, \sigma^2) = \frac{1}{2\sigma^2}\norm{\bar{x} - \mu}^2.
$$
And assuming $\bar{x} = \frac{1}{n} \sum_{i=1}^n x_i$, $x_i \sim \N(\mu, \sigma)$, we have that $\Expect{\bar{x}}{\norm{\bar{x} - \mu}^2} = \sigma^2/n$, giving you a convergence rate of $1/n$.

For a Bayesian approach, assume that you have a prior on $\mu$ given by $\N(\mu_0, \sigma_0^2)$, and that after seeing $n$ samples your posterior is $\N(\mu_n, \sigma_n^2)$. You can consider the convergence in term of expected KL divergence,
$$
\begin{array}{rcl}
\Expect{x \sim \N(\mu_n, \sigma_n^2)}{\KL(\N(x, \sigma^2) || \N(\mu, \sigma^2))}
&=&
\frac{1}{2\sigma^2}\Expect{x \sim \N(\mu_n, \sigma_n^2)}{\norm{x - \mu}^2},\\
&=&
\frac{1}{2\sigma^2}\norm{\mu_n - \mu}^2 + \sigma_n^2.
\end{array}
$$
Following the bayesian update, seeing $\bar{x}$ across $n$ samples, (see for e.g. here),
$$
\begin{array}{rcl}
\sigma_n^2 & = & \left(\frac{1}{\sigma_0^2} + \frac{n}{\sigma^2}\right)^{-1}\\
\mu_n & = & \sigma_n^2 \left(\frac{\mu_0}{\sigma_0^2} + n \frac{\bar{x}}{\sigma^2}\right)
\end{array}
$$
Plugging this back in and solving 
$\Expect{\bar{x}}{\frac{1}{2\sigma^2}\norm{\mu_n - \mu}^2 + \sigma_n^2}$
should also give you a convergence rate in $O(1/n)$.

For a Multivariate Gaussian with unknown covariance matrix, the math gets more complicated but the same principles apply;


*

*You can use a Normal-inverse-Wishart prior,

*use the bayesian update rule (see this table, under Multivariate normal),

*compute the KL divergence for Gaussians (Wikipedia)
