The term divergence means a function $D$, which, given two probability distributions $P,Q$, assigns a non-negative real number $D(P,Q)$ such that $D(P,Q) = 0$ iff $P(x)=Q(x) \forall x$.

The relative entropy (or the Kullback-Leibler divergence) $$D_f(P,Q) = \sum_x P(x)\log\left(\frac{P(x)}{Q(x)}\right)$$ is a classical example of a divergence function.

For any convex real function $f$ on $(0,\infty)$ with $f(1)=0$, Csiszar (independently Ali and Silvey) proposed a divergence given by $$D_f(P,Q) = \sum_x Q(x)f\left(\frac{P(x)}{Q(x)}\right).$$

When $f(x)=x\log x$, we get the relative entropy.

When $f(x) = \frac{x^\alpha-1}{\alpha-1}, \alpha >0, \alpha\neq 1$, $$D_f(P,Q) = \frac{1}{\alpha-1}\left(\sum_x P(x)^{\alpha}Q(x)^{1-\alpha}-1\right).$$ Can someone give a reference for the actual name of this divergence?

  • $\begingroup$ Rényi divergence? (Ref: pdf) $\endgroup$ – Creosote Sep 19 '15 at 18:34
  • 1
    $\begingroup$ What you've written is tantalisingly close to Rényi divergence, $\frac{1}{\alpha-1}\log\sum_x P(x)^\alpha Q(x)^{1-\alpha}$, especially when $P$ and $Q$ are small. In what context does your quoted divergence appear? $\endgroup$ – Creosote Sep 19 '15 at 22:16
  • $\begingroup$ @Creosote Yes, I know that it is closely related to the Renyi divergence. I would like to know who was the one who first studied the mentioned special class of $f$-divergence $D_f(P,Q)$ with a reference. $\endgroup$ – Ashok Sep 20 '15 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.