1
$\begingroup$

In 3 dimensional space, we have a very intuitive picture of the closeness of two vectors: we simply imagine a space with two points in it, and imagine how close they are to eachother. In this way we can imagine different degrees of closeness.

Mathematically, we think of closeness simply as the euclidean metric:

$$\sqrt{(x_1-x_2)^2+(y_1-y_2)^2+(z_1-z_2)^2}$$


In machine learning, we often have sparse features (e.g. word ID's in natural language processing), and we want to embed them in a dense vector. The resulting embedded feature has a conception of closeness, in the sense that any given word ID (e.g. cat) will have a meaning (represented by an embedding) that is closer to a particular word (e.g. feline), than to some other word (car).

We could measure this closeness using the same Euclidean metric on the $n$ entries of the embedded vector:

$$\sqrt {\sum _i (x_{i,1}-x_{i,2})^2}$$


How to visualize $n$ dimensional space in a way that shows intuitively closeness of vectors?

However, there is no longer a straightforward way to visualize these vectors in a way that shows their closeness, since we cannot visualize $n$ dimensional space.

The way I usually mentally visualize $n$ dimensional space, is as a bar chart where each bar represents a coordinate of a particular vector in that space. If $n$ is very large, I usually think of a particular vector as being approximated by a kind of highly oscillatory (real-valued) function (that maps the coordinate index to its output value), kind of like an audio file:

enter image description here

The problem with this visualization, is that it does not represent closeness well at all: If we shift this graph by only one unit to the right, it will look almost indistinguishible from the original graph, but for most graphs, its distance to the original will be enormous. (to see why, imagine in 2 dimensional space we have a vector $(x,y)=(1,10)$. If we shift it by one unit, we get the vector $(10,1)$, a mirror around the diagonal line. The larger the original distance from the origin is, the larger the distance between the new and the original vector will be).


My question is, do we have a way to visualize $n$ dimensional vectors in a way that intuitively shows the closeness/distance of two vectors? (in particular in the context of embeddings of sparse features for machine learning).

$\endgroup$
  • 1
    $\begingroup$ 2300 years ago Euclid taught us that two vectors ("points"), together with their common origin (also a "point"), determine a plane. Therefore you don't even need three dimensions to visualize or reason about closeness of vectors, even in infinite dimensional vector spaces: you only need two dimensions. Generally, to visualize a finite collection of vectors, you only need as many dimensions as they span--and often that can be reduced to a series of two- or (at most) three-dimensional pictures. See stats.stackexchange.com/a/113207/919 for diagrams and more explanation. $\endgroup$ – whuber Jun 3 '17 at 15:19
  • $\begingroup$ I understand that, but then you lose the intuitive picture of the space in which the vectors are embedded. If this is the approach, you might as well just list the distance between two vectors, right? You lose the advantage of visualizing them in the first place, which is that it gives an intuitive view of the vectors. $\endgroup$ – user56834 Jun 3 '17 at 15:21
  • $\begingroup$ Not at all! As the linked post demonstrates, the picture reveals more than just distances. As far as "intuition" goes, if by that you mean "something I'm familiar and comfortable with," I can't help you; but if you understand intuition to mean an analogical way of thinking that can be developed and improved, then there is no inherent contradiction in using multiple ways of visualizing a vector: as a waveform, a bar chart, a point in a plane, a Fourier series, or anything else. For many people, intuition arrives when they learn how to relate those multiple representations. $\endgroup$ – whuber Jun 3 '17 at 15:33
  • $\begingroup$ Converting space to 2 or 3 prime components will show you closeness in almost all cases. $\endgroup$ – keiv.fly Jun 3 '17 at 15:38
  • $\begingroup$ It will show you closeness, but it won't simultaneously show you the entire vector. That is kind of what I'm looking for I suppose. Look at it like this: When you put 2 vectors on a plane (or a line would even be possible), you can see how far they are apart, but if you don't get an overview of their position in the space, which you get with a bar chart/audio-like graph. $\endgroup$ – user56834 Jun 3 '17 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.