Measure of distance between distributions or variables, such as Euclidean distance between points in n-space.

Mathematically a distance, $d$, or metric, is a function that satisfies the following properties. For two points $x, y, z$:

  1. $d(x,y) \geq 0$
  2. $d(x,y) = 0 \implies x = y$
  3. $d(x,y) = d(y,x)$
  4. $d(x,z) \leq d(x,y) + d(y,z)$

Note that certain concepts of distance in probability theory do not satisfy these properties. In particular, the KL-distance between two distributions is not symmetric, and doesn't satisfy the third property above.

Euclidean distance, Manhattan distance and Hamming distance are all common metrics.