$\phi$-divergence?

I am frustrated of looking for a simple explanation of this term $\phi$-divergence, but I cannot find any. Therefore I would be really grateful if somebody could introduce a reference or write a definition (even qualitivative for it). Here is the paper that I first noticed the term in:

Jiménez, R., and J. E. Yukich. "Statistical distances based on Euclidean graphs." Recent Advances in Applied Probability. Springer US, 2005. 223-239.

This paper by Love and Bayraksan gives this introduction: '$\phi$-divergences are used in statistics to measure the "distance" between two distributions,' and this formal definition:

$I_{\phi}(p,q) = \sum_{\omega = 1}^{n}{q_\omega\phi(\frac{p_\omega}{q_\omega})}$

where $p, q$ are probability vectors and $\phi$ is a function on $t > 0$. It cites as examples Kullback-Leibler divergence and $\chi^2$ distance. The introduction begins on p. 5, examples given on p. 7.

Ok found it myself. It was part of the following paper:

Jiménez-Gamero, María-Dolores, et al. "Minimum $\phi$-divergence estimation in misspecified multinomial models." Computational Statistics & Data Analysis 55.12 (2011): 3365-3378.

$\phi: [0,\infty) \to \mathbb{R}$ is a strictly convex function, twice continuously differentiable in $(0,\infty)$. For arbitrary $Q = (q_1,q_2, ... .,q_m)^t,P = (p_1,p_2, . . . ,p_m)^t ∈ \Delta_m$, the $\phi$-divergence between $Q$ and $P$ is defined by (Csiszár, 1967) $$D_\phi(Q,P) =\sum\limits_{i=1}^{m}p_i\phi(\frac{q_i}{p_i}).$$

• +1 For instance, $\phi(x)=-\log(x)$ gives the Kullback-Leibler divergence and $\phi(x)=(1-x)^2$ gives the $\chi^2$ "distance" based on $(p-q)^2/p$. It is noteworthy that $-p\log(q/p)$ $=p-q+\frac{1}{2}(p-q)^2/p+O(1-\frac{q}{p})^3$ (with the linear term $p-q$ vanishing upon summation), thereby connecting those two. – whuber Apr 28 '14 at 21:42
• Thank you very much for the examples. I think your second example is KL! – Cupitor Apr 28 '14 at 22:03
• Yes, it is: the point is that it also equals a constant ($1/2$) times the $\chi^2$ divergence up through second order in $1-\frac{q}{p}$. In fact, the same kind of calculation shows that all $\phi$ divergences are linearly related to one another through second order, assuming only that $\phi$ is twice differentiable in a neighborhood of $1$. – whuber Apr 28 '14 at 22:10
• Wow! Interesting. Where are you getting these facts from? :) – Cupitor Apr 28 '14 at 22:34
• Mathematical calculations :-). I thought you might find them of some relevance because you had asked for explanation, not just a mere definition, and these relationships helped me (at least) get a more intuitive grip on what a $\phi$-divergence is trying to do. I am glad you find them interesting too. – whuber Apr 28 '14 at 22:44

According to Jager and Wellner this term was invented by I. Csiszar (1963), but his paper is in german. Jager and Wellner use it to construct goodness-of-fit tests and they give the following definition:

$K(u,v) = v \phi(\frac{u}{v}) + (1-v)\phi(\frac{1-u}{1-v})$

• Could you indicate where in the Jager and Wellner paper a full definition is given? As far as I can tell with a quick skim, they do not provide a definition but merely refer back to the Csiszar paper. – whuber Apr 28 '14 at 20:59
• Bottom of page 3. I've also edited my answer. – Amit Moscovich Apr 28 '14 at 22:09