So I was reading the Google's paper on VQ-VAE and have stumbled upon the derivation of KL divergence of the uniform prior and the given distribution:

$$q(z=k \mid x)=\left\{\begin{array}{ll}{1} & \text{for } k=\operatorname{argmin}_j \left\|z_e(x)-e_j\right\|_2 \\ 0 & \text{otherwise}\end{array}\right.$$

In the paper it is stated that the KL divergence of the distributions is equal to $\log K$. I understand that the KL is constant but how the $\log K$ derived is pretty unclear to me. Also if the $q(z|x)$ is a one-hot vector then how the Kullback–Leibler distance is even calculated if the distribution contains zero elements. I know we can smooth the distribution bu still.

Here is the link to the paper: arxiv.org/pdf/1711.00937.


The uniform prior is $p(z)=\frac{1}{K}$, for $k=1,\ldots,K$. Then, the KL divergence is $$\begin{align}D_{KL}(q(z\mid x)\mid p(z))&=\underbrace{\sum_{z\in \mathcal{Z}} q(z\mid x)\log\left({q(z\mid x)\over p(z)}\right)}_{\mathcal{Z}=\{z\mid q(z\mid x)\neq 0\}} = \underbrace{q(k\mid x)\log\left({q(k\mid x)\over p(k)}\right)}_{q(k\mid x)\neq 0}\\&=1\cdot\log\left({1\over (1/K)}\right)=\log K\end{align}$$

| cite | improve this answer | |
  • $\begingroup$ Thanks! Also where can I read about the method of constraining the set to avoid the zero values in distribution. Why it gives the same results? Any proof I can find? $\endgroup$ – user2660964 Jul 7 '19 at 16:49
  • $\begingroup$ If you're talking about why I took the $z$ values where $q(z|x)\neq 0$, not others, it's also written in wiki, definition section, 2nd paragraph. We take the support of the first argument in KL. $\endgroup$ – gunes Jul 7 '19 at 17:10
  • $\begingroup$ Thanks, I see it now. Should read the properties then. $\endgroup$ – user2660964 Jul 7 '19 at 18:08
  • $\begingroup$ Regarding the stop-gradient operator in VQ-VAE: stats.stackexchange.com/questions/420562/… $\endgroup$ – root Aug 4 '19 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.