Alternate (?) definition of sample variance [duplicate]

Question

The variance of a sample can be defined as $$s^2 = \frac{1}{2}\frac{1}{n(n-1)}\sum_{i}\sum_{j\ne i}\left(x_i - x_j\right)^2$$

Apart from the factor of $1/2$, this can be paraphrased verbally as

The variance is the average of the squared distances between pairs of distinct data points

Mathematically, this is equivalent to the "usual" definition of variance, $s^2 = \frac{1}{n-1}\sum \left(x_i -\bar{x} \right)^2$. Conceptually, however, it seems (to me) quite different, in two respects:

1. This definition makes no reference to the mean value $\bar{x}$; we are not measuring how far apart points are from the mean, but rather how far away points are from one another.
2. The factor of $n-1$ in the denominator -- which is well-known to be a source of confusion for students (see, e.g., Intuitive explanation for dividing by $n-1$ when calculating standard deviation?) -- appears naturally because there are $n(n-1)$ ordered pairs of distinct data points $(x_i, x_j), i\ne j$. No need for either hand-waving justifications about "cushioning" the sample variance, or for complicated calculations of estimator bias.

Again, just to be clear, I understand why the standard definition of $s^2$ includes a denominator of $n-1$, and I understand that the double-sum definition above is mathematically equivalent to the standard definition. What I would like to know:

Are there contexts (pedagogical or otherwise) in which the alternate definition of variance (as a double-sum over pairs of data points) is more commonly referred to? Are there any textbooks, for example, that take this as the primary definition?

Answer 1

Using symmetry, your double sum can be written as $$ s^2 = \text{ave}_{i < j} f(x_i, x_j) $$ with $f(x, y) := \frac{1}{2} (x - y)^2$ and as such is a U-statistic of degree 2 with kernel $f$ for the estimation of the parameter $\theta = E(f(X, Y)) = \ldots = \text{Var}(X)$, $X$, $Y$ iid. It is actually one of the most prominent, non-trivial examples of a U-statistic and shines accordingly in theoretical math stats classes.

Being a U-statistic is nice because e.g.

• a U-statistic is automatically "optimal" in the sense of being U-nbiased with minimum variance (bye-bye Rao-Blackwell theorem) and

• there is a (more or less explicit) formula for its variance and thus its standard error.

Nevertheless, this alternative representation of the sample variance $s^2$ is rarely used in practice although, as you mentioned, it seems to highlight a different aspect of variation ("typical squared difference between two random picks" instead of "typical squared difference between random pick and mean").

The U-statistics representation of $s^2$ always reminds me of Gini impurity used in decision tree learning where that often plays the role of variance for categorical responses. For a discrete random variable $Z$ with levels $z_1, \dots, z_m$ and $\text{Pr}(Z = z_j) = p_j$, it is defined as $$ I(Z) = 1 - \sum_{j = 1}^m p_i^2 = \sum_{i}\sum_{j \ne i} p_i p_j, $$ and as such is the probability of two random picks not being equal (again something like a typical difference between two random picks).