4
$\begingroup$

I am frustrated of looking for a simple explanation of this term $\phi$-divergence, but I cannot find any. Therefore I would be really grateful if somebody could introduce a reference or write a definition (even qualitivative for it). Here is the paper that I first noticed the term in:

Jiménez, R., and J. E. Yukich. "Statistical distances based on Euclidean graphs." Recent Advances in Applied Probability. Springer US, 2005. 223-239.

Thanks in advance.

$\endgroup$
2
$\begingroup$

This paper by Love and Bayraksan gives this introduction: '$\phi$-divergences are used in statistics to measure the "distance" between two distributions,' and this formal definition:

$I_{\phi}(p,q) = \sum_{\omega = 1}^{n}{q_\omega\phi(\frac{p_\omega}{q_\omega})}$

where $p, q$ are probability vectors and $\phi$ is a function on $t > 0$. It cites as examples Kullback-Leibler divergence and $\chi^2$ distance. The introduction begins on p. 5, examples given on p. 7.

$\endgroup$
2
$\begingroup$

Ok found it myself. It was part of the following paper:

Jiménez-Gamero, María-Dolores, et al. "Minimum $\phi$-divergence estimation in misspecified multinomial models." Computational Statistics & Data Analysis 55.12 (2011): 3365-3378.

$\phi: [0,\infty) \to \mathbb{R}$ is a strictly convex function, twice continuously differentiable in $(0,\infty)$. For arbitrary $Q = (q_1,q_2, ... .,q_m)^t,P = (p_1,p_2, . . . ,p_m)^t ∈ \Delta_m$, the $\phi$-divergence between $Q$ and $P$ is defined by (Csiszár, 1967) $$D_\phi(Q,P) =\sum\limits_{i=1}^{m}p_i\phi(\frac{q_i}{p_i}).$$

$\endgroup$
  • 2
    $\begingroup$ +1 For instance, $\phi(x)=-\log(x)$ gives the Kullback-Leibler divergence and $\phi(x)=(1-x)^2$ gives the $\chi^2$ "distance" based on $(p-q)^2/p$. It is noteworthy that $-p\log(q/p) $ $=p-q+\frac{1}{2}(p-q)^2/p+O(1-\frac{q}{p})^3$ (with the linear term $p-q$ vanishing upon summation), thereby connecting those two. $\endgroup$ – whuber Apr 28 '14 at 21:42
  • $\begingroup$ Thank you very much for the examples. I think your second example is KL! $\endgroup$ – Cupitor Apr 28 '14 at 22:03
  • 2
    $\begingroup$ Yes, it is: the point is that it also equals a constant ($1/2$) times the $\chi^2$ divergence up through second order in $1-\frac{q}{p}$. In fact, the same kind of calculation shows that all $\phi$ divergences are linearly related to one another through second order, assuming only that $\phi$ is twice differentiable in a neighborhood of $1$. $\endgroup$ – whuber Apr 28 '14 at 22:10
  • $\begingroup$ Wow! Interesting. Where are you getting these facts from? :) $\endgroup$ – Cupitor Apr 28 '14 at 22:34
  • 1
    $\begingroup$ Mathematical calculations :-). I thought you might find them of some relevance because you had asked for explanation, not just a mere definition, and these relationships helped me (at least) get a more intuitive grip on what a $\phi$-divergence is trying to do. I am glad you find them interesting too. $\endgroup$ – whuber Apr 28 '14 at 22:44
1
$\begingroup$

According to Jager and Wellner this term was invented by I. Csiszar (1963), but his paper is in german. Jager and Wellner use it to construct goodness-of-fit tests and they give the following definition:

$K(u,v) = v \phi(\frac{u}{v}) + (1-v)\phi(\frac{1-u}{1-v})$

$\endgroup$
  • 1
    $\begingroup$ Could you indicate where in the Jager and Wellner paper a full definition is given? As far as I can tell with a quick skim, they do not provide a definition but merely refer back to the Csiszar paper. $\endgroup$ – whuber Apr 28 '14 at 20:59
  • $\begingroup$ Bottom of page 3. I've also edited my answer. $\endgroup$ – Amit Moscovich Apr 28 '14 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.