# $\phi$-divergence?

I am frustrated of looking for a simple explanation of this term $$\phi$$-divergence, but I cannot find any. Therefore I would be really grateful if somebody could introduce a reference or write a definition (even qualitative for it). Here is the paper that I first noticed the term in:

Jiménez, R., and J. E. Yukich. "Statistical distances based on Euclidean graphs." Recent Advances in Applied Probability. Springer US, 2005. 223-239.

• $\mathcal{D}_\varphi\stackrel{\small \text{def.}}{=}\int_\mathcal{x}\big(\frac{\rm d\alpha}{\rm d\beta}\big){\rm d}\beta +\varphi'_\infty\alpha^\bot(\mathcal{x})$ More details are in chapter 8, Computational Optimal Transport. Mar 2, 2020 at 2:18

Ok found it myself. It was part of the following paper:

Jiménez-Gamero, María-Dolores, et al. "Minimum $\phi$-divergence estimation in misspecified multinomial models." Computational Statistics & Data Analysis 55.12 (2011): 3365-3378.

$\phi: [0,\infty) \to \mathbb{R}$ is a strictly convex function, twice continuously differentiable in $(0,\infty)$. For arbitrary $Q = (q_1,q_2, ... .,q_m)^t,P = (p_1,p_2, . . . ,p_m)^t ∈ \Delta_m$, the $\phi$-divergence between $Q$ and $P$ is defined by (Csiszár, 1967) $$D_\phi(Q,P) =\sum\limits_{i=1}^{m}p_i\phi(\frac{q_i}{p_i}).$$

• +1 For instance, $\phi(x)=-\log(x)$ gives the Kullback-Leibler divergence and $\phi(x)=(1-x)^2$ gives the $\chi^2$ "distance" based on $(p-q)^2/p$. It is noteworthy that $-p\log(q/p)$ $=p-q+\frac{1}{2}(p-q)^2/p+O(1-\frac{q}{p})^3$ (with the linear term $p-q$ vanishing upon summation), thereby connecting those two.
– whuber
Apr 28, 2014 at 21:42
• Thank you very much for the examples. I think your second example is KL! Apr 28, 2014 at 22:03
• Yes, it is: the point is that it also equals a constant ($1/2$) times the $\chi^2$ divergence up through second order in $1-\frac{q}{p}$. In fact, the same kind of calculation shows that all $\phi$ divergences are linearly related to one another through second order, assuming only that $\phi$ is twice differentiable in a neighborhood of $1$.
– whuber
Apr 28, 2014 at 22:10
• Wow! Interesting. Where are you getting these facts from? :) Apr 28, 2014 at 22:34
• Mathematical calculations :-). I thought you might find them of some relevance because you had asked for explanation, not just a mere definition, and these relationships helped me (at least) get a more intuitive grip on what a $\phi$-divergence is trying to do. I am glad you find them interesting too.
– whuber
Apr 28, 2014 at 22:44

This paper by Love and Bayraksan gives this introduction: '$\phi$-divergences are used in statistics to measure the "distance" between two distributions,' and this formal definition:

$I_{\phi}(p,q) = \sum_{\omega = 1}^{n}{q_\omega\phi(\frac{p_\omega}{q_\omega})}$

where $p, q$ are probability vectors and $\phi$ is a function on $t > 0$. It cites as examples Kullback-Leibler divergence and $\chi^2$ distance. The introduction begins on p. 5, examples given on p. 7.

According to Jager and Wellner this term was invented by I. Csiszar (1963), but his paper is in german. Jager and Wellner use it to construct goodness-of-fit tests and they give the following definition:

$K(u,v) = v \phi(\frac{u}{v}) + (1-v)\phi(\frac{1-u}{1-v})$

• Could you indicate where in the Jager and Wellner paper a full definition is given? As far as I can tell with a quick skim, they do not provide a definition but merely refer back to the Csiszar paper.
– whuber
Apr 28, 2014 at 20:59
• Bottom of page 3. I've also edited my answer. Apr 28, 2014 at 22:09