2
$\begingroup$

In 3 dimensional space, we have a very intuitive picture of the closeness of two vectors: we simply imagine a space with two points in it, and imagine how close they are to eachother. In this way we can imagine different degrees of closeness.

Mathematically, we think of closeness simply as the euclidean metric:

$$\sqrt{(x_1-x_2)^2+(y_1-y_2)^2+(z_1-z_2)^2}$$

In machine learning, we often have sparse features (e.g. word ID's in natural language processing), and we want to embed them in a dense vector. The resulting embedded feature has a conception of closeness, in the sense that any given word ID (e.g. cat) will have a meaning (represented by an embedding) that is closer to a particular word (e.g. feline), than to some other word (car).

We could measure this closeness using the same Euclidean metric on the $n$ entries of the embedded vector:

$$\sqrt {\sum _i (x_{i,1}-x_{i,2})^2}$$

How to visualize $n$ dimensional space in a way that shows intuitively closeness of vectors?

However, there is no longer a straightforward way to visualize these vectors in a way that shows their closeness, since we cannot visualize $n$ dimensional space.

The way I usually mentally visualize $n$ dimensional space, is as a bar chart where each bar represents a coordinate of a particular vector in that space. If $n$ is very large, I usually think of a particular vector as being approximated by a kind of highly oscillatory (real-valued) function (that maps the coordinate index to its output value), kind of like an audio file:

enter image description here

The problem with this visualization, is that it does not represent closeness well at all: If we shift this graph by only one unit to the right, it will look almost indistinguishible from the original graph, but for most graphs, its distance to the original will be enormous. (to see why, imagine in 2 dimensional space we have a vector $(x,y)=(1,10)$. If we shift it by one unit, we get the vector $(10,1)$, a mirror around the diagonal line. The larger the original distance from the origin is, the larger the distance between the new and the original vector will be).

My question is, do we have a way to visualize $n$ dimensional vectors in a way that intuitively shows the closeness/distance of two vectors? (in particular in the context of embeddings of sparse features for machine learning).

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ 2300 years ago Euclid taught us that two vectors ("points"), together with their common origin (also a "point"), determine a plane. Therefore you don't even need three dimensions to visualize or reason about closeness of vectors, even in infinite dimensional vector spaces: you only need two dimensions. Generally, to visualize a finite collection of vectors, you only need as many dimensions as they span--and often that can be reduced to a series of two- or (at most) three-dimensional pictures. See stats.stackexchange.com/a/113207/919 for diagrams and more explanation. $\endgroup$
    – whuber ♦
    Jun 3, 2017 at 15:19
  • $\begingroup$ I understand that, but then you lose the intuitive picture of the space in which the vectors are embedded. If this is the approach, you might as well just list the distance between two vectors, right? You lose the advantage of visualizing them in the first place, which is that it gives an intuitive view of the vectors. $\endgroup$
    – user56834
    Jun 3, 2017 at 15:21
  • $\begingroup$ Not at all! As the linked post demonstrates, the picture reveals more than just distances. As far as "intuition" goes, if by that you mean "something I'm familiar and comfortable with," I can't help you; but if you understand intuition to mean an analogical way of thinking that can be developed and improved, then there is no inherent contradiction in using multiple ways of visualizing a vector: as a waveform, a bar chart, a point in a plane, a Fourier series, or anything else. For many people, intuition arrives when they learn how to relate those multiple representations. $\endgroup$
    – whuber ♦
    Jun 3, 2017 at 15:33
  • $\begingroup$ Converting space to 2 or 3 prime components will show you closeness in almost all cases. $\endgroup$
    – keiv.fly
    Jun 3, 2017 at 15:38
  • $\begingroup$ It will show you closeness, but it won't simultaneously show you the entire vector. That is kind of what I'm looking for I suppose. Look at it like this: When you put 2 vectors on a plane (or a line would even be possible), you can see how far they are apart, but if you don't get an overview of their position in the space, which you get with a bar chart/audio-like graph. $\endgroup$
    – user56834
    Jun 3, 2017 at 17:30

1 Answer 1

Reset to default
1
$\begingroup$

Euclidean distances are not a very good metric for quantifying n-space distances. For example, consider this work: "To avoid the undesired effects of distance concentration in high-dimensional spaces, previous work has already advocated the use of fractional ℓp norms instead of the ubiquitous Euclidean norm. Closely related to concentration is the emergence of hub and anti-hub objects. Hub objects have a small distance to an exceptionally large number of data points while anti-hubs lie far from all other data points. The contribution of this work is an empirical examination of concentration and hubness, resulting in an unsupervised approach for choosing an ℓp norm by minimizing hubs while simultaneously maximizing nearest neighbor classification."

Which is re machine-learning so it is on-topic. I would suggest that before you worry excessively on visualization of n-space objects, that you select a metric for proximity that is somewhat more telling.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.