We can compute the Euclidean distance between two vectors $\mathbf{x}$ and $\mathbf{y}$ by: $$ d(\mathbf{x}, \mathbf{y}) = \sqrt{(x_1-y_1)^2 + \ldots + (x_n - y_n)^2} $$

When there are missing values in either $\mathbf{x}$ or $\mathbf{y}$, what R does, according to the help page, is:

Missing values are allowed, and are excluded from all computations involving the rows within which they occur. Further, when Inf values are involved, all pairs of values are excluded when their contribution to the distance gave NaN or NA. If some columns are excluded in calculating a Euclidean, Manhattan, Canberra or Minkowski distance, the sum is scaled up proportionally to the number of columns used.

This means, in R code:

na.index <- is.na(X) | is.na(Y)
dist(rbind(X[!na.index], Y[!na.index])) * sqrt(length(X) / length(X[!na.idx]))

Which is equivalent to:

dist(rbind(X[!na.index], Y[!na.index])) * sqrt(length(Y) / length(Y[!na.idx]))

I would like to understand what the "proportional scaling" term


is doing. I see that if there are more missing values, the denominator will be smaller and the term is larger, "penalizing" more the distance measure. What I understand is that if we consider the first part of the computation (distance without missing values) as a constant, and focus on the scaling part, and consider the length of the vector to be 1 and the proportion of availability $a$ to be any number between $[0, 1]$, then the "scaling factor" would behave like $\sqrt{1/a}$:

Plot for sqrt(1/x)

The two questions are: 1. is this reasoning correct?, and 2. does it make sense to use this scaling factor for situations where the vectors are binary (0/1) and using a binary similarity/distance measure, like with the Jaccard index? If not, what are good ways to account for missing data when calculating similarities/distances with binary measures?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.