8
$\begingroup$

There exist a number of tools that provide a distance between two continuous probability distributions. Most (semi)distances, like the Kullback-Leibler divergence, use probability density functions. However, the literature is quite sparse when it comes to comparing two distributions in Fourier space, i.e. via their characteristic function. Is there an elegant way to do so?

$\endgroup$
  • 1
    $\begingroup$ I thought of using spectral coherence (en.wikipedia.org/wiki/Coherence_%28signal_processing%29) but do not know how to apply it to this case $\endgroup$ – yannick May 6 '14 at 17:33
  • $\begingroup$ Did you find an answer to this? $\endgroup$ – becko Apr 5 '18 at 18:25
  • $\begingroup$ Google for "empirical characteristic function", it can be used for construction hypothesis tests, and certainly for constructing distances. $\endgroup$ – kjetil b halvorsen Apr 8 '18 at 15:09
1
$\begingroup$

One notable distance that can be considered in Fourier space is the maximum mean discrepancy (MMD). One first selects a positive semi-definite kernel $k : \mathcal X \times \mathcal X \to \mathbb R$ corresponding to a reproducing kernel Hilbert space (RKHS) $\mathcal H_k$. The MMD is then

\begin{align} \operatorname{MMD}_k(\mathbb P, \mathbb Q) &= \sup_{f \in \mathcal H_k : \lVert f \rVert_{\mathcal H_k} \le 1} \mathbb E_{X \sim \mathbb P}[ f(X)] - \mathbb E_{Y \sim \mathbb Q}[ f(Y)] \\&= \left\lVert \mathbb E_{X \sim \mathbb P}[k(X, \cdot)] - \mathbb E_{Y \sim \mathbb Q}[k(Y, \cdot)] \right\rVert \\&= \sqrt{ \mathbb{E}_{\substack{X, X' \sim \mathbb P\\Y, Y' \sim \mathbb Q}}\left[ k(X, X') + k(Y, Y') - 2 k(X, Y) \right] } .\end{align} You might be familiar with the energy distance; that is a special case of the MMD for a particular choice of kernel. This is a proper metric for many choices of kernel, called characteristic kernels; it is always a semimetric.

What does this have to do with the Fourier transform? Well, if $\mathcal X = \mathbb R^d$ and $k(x, y) = \psi(x - y)$, so that $\psi : \mathbb R^d \to \mathbb R$ is a positive-definite function, then it turns out the MMD can also be written as $$ \operatorname{MMD}_k(\mathbb P, \mathbb Q) = \sqrt{\int \left\lvert \varphi_{\mathbb P}(\omega) - \varphi_{\mathbb Q}(\omega) \right\rvert^2 \, \mathrm{d}\hat\psi(\omega)} $$ where $\varphi$ denotes the characteristic function, and $\hat\psi$ is the Fourier transform of $\psi$ in the measure sense. (It will always be a finite nonnegative measure; you can see from this definition that a translation-invariant kernel is characteristic iff its Fourier transform is everywhere positive.) For a proof, see Corollary 4 of

Sriperumbudur et al., Hilbert space embeddings and metrics on probability measures, JMLR 2010.

The MMD – which you can easily estimate via the third form above – thus compares distributions by the $L_2$ distance between their full characteristic functions, with frequencies weighted according to the choice of kernel. For example, the common Gaussian kernel $k(x, y) = \exp\left( -\frac{1}{2\sigma^2} \lVert x - y \rVert^2 \right)$ will weight the frequencies with a Gaussian with mean 0 and variance $1/\sigma^2$.

Sometimes it's better, and sometimes computationally faster / more informative, to instead compare the characteristic functions at particular locations rather than everywhere. It turns out it's better to make a slight tweak, evaluating differences in smoothed characteristic functions at random locations:

Chwialkowski et al., Fast Two-Sample Testing with Analytic Representations of Probability Measures, NeurIPS 2015.

A follow-up work finds the most informative frequencies to test, rather than random:

Jitkrittum et al., Interpretable Distribution Features with Maximum Testing Power, NeurIPS 2016.

These are all closely connected to classical tests based on empircal characteristic functions as mentioned by kjetil in the comments.

$\endgroup$
  • $\begingroup$ Great reply to a question I asked almost 6 years ago! $\endgroup$ – yannick Jan 10 at 11:37
  • $\begingroup$ @yannick It popped up in the related bar, figured I could share for posterity. :) $\endgroup$ – Dougal Jan 10 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.