8
$\begingroup$

I am frustrated of looking for a simple explanation of this term $\phi$-divergence, but I cannot find any. Therefore I would be really grateful if somebody could introduce a reference or write a definition (even qualitative for it). Here is the paper that I first noticed the term in:

Jiménez, R., and J. E. Yukich. "Statistical distances based on Euclidean graphs." Recent Advances in Applied Probability. Springer US, 2005. 223-239.

Improve this question
$\endgroup$
1
  • $\begingroup$ $\mathcal{D}_\varphi\stackrel{\small \text{def.}}{=}\int_\mathcal{x}\big(\frac{\rm d\alpha}{\rm d\beta}\big){\rm d}\beta +\varphi'_\infty\alpha^\bot(\mathcal{x})$ More details are in chapter 8, Computational Optimal Transport. $\endgroup$
    – ZuoyanLi
    Mar 2, 2020 at 2:18

3 Answers 3

Reset to default
6
$\begingroup$

Ok found it myself. It was part of the following paper:

Jiménez-Gamero, María-Dolores, et al. "Minimum $\phi$-divergence estimation in misspecified multinomial models." Computational Statistics & Data Analysis 55.12 (2011): 3365-3378.

$\phi: [0,\infty) \to \mathbb{R}$ is a strictly convex function, twice continuously differentiable in $(0,\infty)$. For arbitrary $Q = (q_1,q_2, ... .,q_m)^t,P = (p_1,p_2, . . . ,p_m)^t ∈ \Delta_m$, the $\phi$-divergence between $Q$ and $P$ is defined by (Csiszár, 1967) $$D_\phi(Q,P) =\sum\limits_{i=1}^{m}p_i\phi(\frac{q_i}{p_i}).$$

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ +1 For instance, $\phi(x)=-\log(x)$ gives the Kullback-Leibler divergence and $\phi(x)=(1-x)^2$ gives the $\chi^2$ "distance" based on $(p-q)^2/p$. It is noteworthy that $-p\log(q/p) $ $=p-q+\frac{1}{2}(p-q)^2/p+O(1-\frac{q}{p})^3$ (with the linear term $p-q$ vanishing upon summation), thereby connecting those two. $\endgroup$
    – whuber ♦
    Apr 28, 2014 at 21:42
  • $\begingroup$ Thank you very much for the examples. I think your second example is KL! $\endgroup$
    – Cupitor
    Apr 28, 2014 at 22:03
  • 2
    $\begingroup$ Yes, it is: the point is that it also equals a constant ($1/2$) times the $\chi^2$ divergence up through second order in $1-\frac{q}{p}$. In fact, the same kind of calculation shows that all $\phi$ divergences are linearly related to one another through second order, assuming only that $\phi$ is twice differentiable in a neighborhood of $1$. $\endgroup$
    – whuber ♦
    Apr 28, 2014 at 22:10
  • $\begingroup$ Wow! Interesting. Where are you getting these facts from? :) $\endgroup$
    – Cupitor
    Apr 28, 2014 at 22:34
  • 1
    $\begingroup$ Mathematical calculations :-). I thought you might find them of some relevance because you had asked for explanation, not just a mere definition, and these relationships helped me (at least) get a more intuitive grip on what a $\phi$-divergence is trying to do. I am glad you find them interesting too. $\endgroup$
    – whuber ♦
    Apr 28, 2014 at 22:44
3
$\begingroup$

This paper by Love and Bayraksan gives this introduction: '$\phi$-divergences are used in statistics to measure the "distance" between two distributions,' and this formal definition:

$I_{\phi}(p,q) = \sum_{\omega = 1}^{n}{q_\omega\phi(\frac{p_\omega}{q_\omega})}$

where $p, q$ are probability vectors and $\phi$ is a function on $t > 0$. It cites as examples Kullback-Leibler divergence and $\chi^2$ distance. The introduction begins on p. 5, examples given on p. 7.

Improve this answer
$\endgroup$
1
$\begingroup$

According to Jager and Wellner this term was invented by I. Csiszar (1963), but his paper is in german. Jager and Wellner use it to construct goodness-of-fit tests and they give the following definition:

$K(u,v) = v \phi(\frac{u}{v}) + (1-v)\phi(\frac{1-u}{1-v})$

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Could you indicate where in the Jager and Wellner paper a full definition is given? As far as I can tell with a quick skim, they do not provide a definition but merely refer back to the Csiszar paper. $\endgroup$
    – whuber ♦
    Apr 28, 2014 at 20:59
  • $\begingroup$ Bottom of page 3. I've also edited my answer. $\endgroup$
    – Amit Moscovich
    Apr 28, 2014 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.