# Why is the importance of variable used in clustering proportional to its variance? Equation 14.27 ESL

Basically, I don't understand how the variance was derived here:

$$\overline{d}_j = \frac{1}{N^2}\sum_{i=1}^N\sum_{i^{'}=1}^N(x_{ij}-x_{i^{'}j})^2 = 2 \cdot var_j$$

That is an equation 14.27 from Elements of Statistical Learning.

Excerpt from section 14.3.3:

The influence of the jth attribute Xj on object dissimilarity D(xi , xi′ ) depends upon its relative contribution to the average object dissimilarity measure over all pairs of observations in the data set $$\overline{D} = \frac{1}{N^2}\sum_{i=1}^N\sum_{i^{'}=1}^N D(x_{i},x_{i^{'}}) = \sum_{i^{'}=1}^N w_j \cdot \overline{d_j}$$ with $$\overline{d}_j = \frac{1}{N^2}\sum_{i=1}^N\sum_{i^{'}=1}^N d(x_{ij},x_{i^{'}j})$$

Using squared-error distance (d(xij, xi'j)) for each coordinate, equation 14.27 can be found.

Easing up the notation, let's say we have a set of values ($$N$$ values) in an array. The following quantity \begin{align}\frac{1}{N^2}\sum_i \sum_j (x_i-x_j)^2&=\frac{1}{N^2}\sum_i\left(\sum_j x_i^2-2x_ix_j+x_j^2\right)\\&=\frac{1}{N} \sum_i x_i^2 -\frac{2}{N^2}\sum_i x_i\sum_j x_j+\frac{1}{N}\sum_j x_j^2\\&=\frac{2}{N}\sum_i x_i^2 - 2\bar{x^2}=2\overline {x^2}-2\bar x^2 = \widehat{2\operatorname{var}(x)}\end{align}
I have ignored $$j$$ in your notation as it doesn't contribute to anything, and changed $$i'$$ to $$j$$ instead to make notation more clear. I've also ignored the upper and lower limits of the summations, as both are from $$1$$ to $$N$$.
The expression is two times the estimate of population variance because $$\operatorname{var}(X)=E[X^2]-E[X]^2$$