5
$\begingroup$

So I was reading the Google's paper on VQ-VAE and have stumbled upon the derivation of KL divergence of the uniform prior and the given distribution:

$$q(z=k \mid x)=\left\{\begin{array}{ll}{1} & \text{for } k=\operatorname{argmin}_j \left\|z_e(x)-e_j\right\|_2 \\ 0 & \text{otherwise}\end{array}\right.$$

In the paper it is stated that the KL divergence of the distributions is equal to $\log K$. I understand that the KL is constant but how the $\log K$ derived is pretty unclear to me. Also if the $q(z|x)$ is a one-hot vector then how the Kullback–Leibler distance is even calculated if the distribution contains zero elements. I know we can smooth the distribution bu still.

Here is the link to the paper: arxiv.org/pdf/1711.00937.

$\endgroup$

1 Answer 1

5
$\begingroup$

The uniform prior is $p(z)=\frac{1}{K}$, for $k=1,\ldots,K$. Then, the KL divergence is $$\begin{align}D_{KL}(q(z\mid x)\mid p(z))&=\underbrace{\sum_{z\in \mathcal{Z}} q(z\mid x)\log\left({q(z\mid x)\over p(z)}\right)}_{\mathcal{Z}=\{z\mid q(z\mid x)\neq 0\}} = \underbrace{q(k\mid x)\log\left({q(k\mid x)\over p(k)}\right)}_{q(k\mid x)\neq 0}\\&=1\cdot\log\left({1\over (1/K)}\right)=\log K\end{align}$$

$\endgroup$
6
  • $\begingroup$ Thanks! Also where can I read about the method of constraining the set to avoid the zero values in distribution. Why it gives the same results? Any proof I can find? $\endgroup$ Commented Jul 7, 2019 at 16:49
  • 1
    $\begingroup$ If you're talking about why I took the $z$ values where $q(z|x)\neq 0$, not others, it's also written in wiki, definition section, 2nd paragraph. We take the support of the first argument in KL. $\endgroup$
    – gunes
    Commented Jul 7, 2019 at 17:10
  • $\begingroup$ Thanks, I see it now. Should read the properties then. $\endgroup$ Commented Jul 7, 2019 at 18:08
  • $\begingroup$ Regarding the stop-gradient operator in VQ-VAE: stats.stackexchange.com/questions/420562/… $\endgroup$
    – root
    Commented Aug 4, 2019 at 15:58
  • 1
    $\begingroup$ @sfotiadis it is en.wikipedia.org/wiki/… but I believe it has changed since $\endgroup$
    – gunes
    Commented Jul 3, 2022 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.