How to measure distance for features with different scales?

Question

I'm reading the book "Collective Intelligence" and in one chapter they introduce how to measure similarity between users on a movie review website with euclidean distance.

Now are the movies rated all on the scale from 1-5. But what if I want to find similar users based on features like - lets say body height, body width, weight and ratio of eye-distance to nose-length. This features operate on different scales, so e.g. body height would influence the distance much more than the eye-nose ratio.

My question is what is the best way to approach this example. Should one use a different distance measure (which?), or normalize the data somehow and use euclidean distance?

Answer 1

A very common solution for this very common problem (ie, over-weighting variables) is to standardize your data.

To do this, you just perform two successive column-wise operations on your data:

• subtract the mean and

• divide by the standard deviation
  

The rationale of these two operations is to ensure the values have zero-mean (when subtracting the mean in the numerator) and unit-variance (by dividing with the standard deviation in the denominator).

For instance, in NumPy:

>>> # first create a small data matrix comprised of three variables 
>>> # having three different 'scales' (means and variances)

>>> a = 10*NP.random.rand(6)
>>> b = 50*NP.random.rand(6)
>>> c = 2*NP.random.rand(6)
>>> A = NP.column_stack((a, b, c))
>>> A   # the pre-standardized data
    array([[  1.753,  37.809,   1.181],
           [  1.386,   8.333,   0.235],
           [  2.827,  40.5  ,   0.625],
           [  5.516,  47.202,   0.183],
           [  0.599,  27.017,   1.054],
           [  8.918,  35.398,   1.602]])

>>> # mean center the data (columnwise)
>>> A -= NP.mean(A, axis=0)
>>> A
    array([[ -1.747,   5.099,   0.368],
           [ -2.114, -24.377,  -0.578],
           [ -0.673,   7.79 ,  -0.189],
           [  2.016,  14.493,  -0.631],
           [ -2.901,  -5.693,   0.24 ],
           [  5.418,   2.688,   0.789]])

>>> # divide by the standard deviation
>>> A /= NP.std(A, axis=0)
>>> A
    array([[-0.606,  0.409,  0.716],
           [-0.734, -1.957, -1.125],
           [-0.233,  0.626, -0.367],
           [ 0.7  ,  1.164, -1.228],
           [-1.007, -0.457,  0.468],
           [ 1.881,  0.216,  1.536]])

Answer 2

One option is Gower's generalised (dis)simmilarity coefficient. It is defined as

$$s_{ij} = \frac{\sum \limits_{k = 1}^{m} w_{ijk} s_{ijk}}{\sum \limits_{k = 1}^{m} w_{ijk}}$$

where $s_{ij}$ is the overall similarity between samples $i$ and $j$, $s_{ijk}$ is the similarity between $i$ and $j$ for the $k$th variable and $w_{ijk}$ is the weight applied to the $k$th variable for samples $i$ and $j$.

The $s_{ijk}$ are computed separately for each of the $k$ variables, which is what allows the function to work on data in different units. If the data are binary (0/1) or nomial (classes) then $s_{ijk} = 1$ if and only if both $i$ and $j$ are in the same class or (both present or both absent).

For continuous data, $s_{ijk}$ is computed as

$$s_{ijk} = \frac{1 - |x_{ik} - x_{jk}|}{r_k}$$

where $x$ are the observed data for $i$ or $j$ on the $k$th variable and $r_K$ is the range of the $k$th variable. $r_k$ is an implicit standardisation, again accounting for the fact that each variable is in different units.

The weights $w_{ijk}$ allow additional flexibility and also a facility to handle missing data. If the data are missing for $i$ or $j$ on the $k$th variable then that comparison doesn't contribute to the overall similarity between the two variables as that pairing gets weight 0. If the data are available for $x_i$ and $x_j$ for the $k$th variable then $w_{ijk} = 1$. Weights between 0 and 1 allow the user to give weight to different variables.

Answer 3

Despite that there is a huge diversity of proximity measures you could still use euclidean distance in your case. The prerequisite for euclidean distance is interval level of measurement of all variables. All your 4 variables height, width, weight, and the ratio are interval. So, after standardization (such as the one suggested by @doug) of the variables you may apply euclidean distance. In your place, however, I would perhaps consider taking logarithm or angular transformation of the ratio variable first (before standardization). And I don't think you really need Gower coefficient (suggested by @Gavin) in your case, because all your variables are of the same level of measurement.