Why is it called $\chi^2$ distance / kernel?

Question

The $\chi^2$ distance function is defined as

$$ \chi(u,v) = \sum_{i=1}^n \frac{(u_i-v_i)^2}{u_i+v_i} $$

and the $\chi^2$ kernel function, used in support vector machines, is $$ K(u,v) = \exp(-c \chi(u,v) ) $$ for some hyperparameter $c$.

This distance function and kernel are most commonly used to compare the similarities between two histogram samples, e.g. in bag-of-words or bag-of-feature applications.

The name suggests some connection with either the $\chi^2$ distribution, or the $\chi^2$ Pearson test. The closest I can get is that the $\chi^2$ distance is trying to approximate $$ \sum_{i=1}^k \frac{(O_i-E_i)^2}{E_i} $$

where $O_i$ is the number of observed samples in bin $i$ and $E_i$ is the expected number of samples in bin $i$. But, to say that this quantity asymptotically approaches a $\chi^2$ distribution with degree $k-1$, it doesn't seem connected to the distance function or kernel application that much.

Question: What is the connection between the $\chi^2$ distance function, or kernel application, to the $\chi^2$ distribution, especially for degree higher than 1 (and the PDF is not an exponential decay)? Or is there none and it's just a naming idiosyncrasy? Any sources is also appreciated Thank you!

Answer 1

The distance metric is usually attributed to either LeCam or I. Vincze. The reason why people started calling it $\chi^2$ is that it can be seen as "symmetrized Pearson", see this excerpt from On Measures of Entropy and Information, Tech. Note 009 v0.7, http://threeplusone.com/info , Gavin E. Crooks,2018-09-22:

Answer 2

I think it originated from [1], it seems inspired by the $\chi^2$ distance as you said, but it does not have any theoretical motivation.

The average between each feature is used as their expected value. Therefore,

$$ D(x, y) = \sum_i \frac{(x_i - \mu_i)^2 }{\mu_i} $$

where $$ \mu_i = \frac{x_i + y_i}{2} $$

such that $$ D(x, y) = 2 \sum_i \frac{\left(x_i - \frac{x_i + y_i}{2}\right)^2}{x_i + y_i} \\ = \frac{2}{4} \sum_i \frac{(2 x_i - x_i - y_i)^2}{x_i + y_i} \\ = \frac{1}{2} \sum_i \frac{(x_i - y_i)^2}{x_i + y_i} $$

The use with the RBF kernel is seen in [2], but no explanation is given besides that D(x, y) is a metric.

[1] Puzicha, J., Hofmann, T., & Buhmann, J. M. (1997, June). Non-parametric similarity measures for unsupervised texture segmentation and image retrieval. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 267-272). IEEE.

[2] Vedaldi, A., & Zisserman, A. (2012). Efficient additive kernels via explicit feature maps. IEEE transactions on pattern analysis and machine intelligence, 34(3), 480-492.