Word embedding refers to feature learning techniques in natural language processing where words are mapped to vectors of real numbers in a low-dimensional space, the embedding space.

Similar to other feature learning (representation learning) techniques, the word vectors are learned within a neural network classifier (e.g. the weights a hidden layer) or by using dimensionality reduction methods (e.g. LSA / PCA) etc. The collection of learned word vectors is basically a matrix, where each row is the learned representation of a word, and each column corresponds to a learned latent feature.

Now, the literature sometimes refers to the embedding space as to a vector space. And I've been recently challenged about the correctness of using the term vector space in this context.

Is the usage of the term vector space acceptable e.g. in machine learning and math communities?

Do word vectors obtained via word embedding techniques really form a vector space?

Is it even practical to concern about this?


3 Answers 3


A few (weak) arguments against using the term "vector space"

The commutativity property of vector addition does not always hold in semantics. Therefore, this property shouldn't (always) hold in the embedding space either. Thus, the embedding space should not be called a vector space.

E.g. attempt to treat semantic composition as vector addition in the vector space:

$v_{rescue \ dog} = v_{rescue} + v_{dog}$ (a dog which is trained to rescue people)

$v_{dog \ rescue} = v_{dog} + v_{rescue}$ (the operation of saving a dog)

The phrases "rescue dog" and "dog rescue" mean different things, but in our hypothetical vector space, they would (incorrectly) have the same vector representation (due to commutativity).

Similarly for the associativity property.

Regarding the existence of additive inverse - this property is often violated in semantics, as many words are non-opposable (e.g. bee). Therefore, one might say it is wrong if the embedding space satisfies the condition of additive inverse. Thus, the embedding space shouldn't be called a vector space.

And so on.

A few (common-sense) arguments in favour of using the term "vector space"

A misuse of the algebraical term vector space does no harm in this context.

Choosing to model the learned word features and word relationships in a vector space implies, indeed, erroneous assumptions and constraints which are irrelevant to natural language, but this can't be coined as unacceptable. This issue seems related to choosing a model with a high bias, which can't capture well all the particularities of the real-world data (natural language).

Furthermore, the term vector space model has never seemed inappropriate. And word vectors obtained through word embedding techniques are, actually, part of this family of models.

  • 1
    $\begingroup$ Since there is no notion of an additive inverse, but since king-man+woman=queen, do you think we should consider the space $V$ to be an affine space instead of a vector space? In an affine space, we have the operation $(x,y,z)\mapsto x-y+z$, but we do not have an origin. $\endgroup$ Commented Jan 1, 2023 at 22:00
  • $\begingroup$ @JosephVanName Precisely correct. $\endgroup$
    – tparker
    Commented Aug 29, 2023 at 2:38

No, they don't. They form a mathematical structure that's much more closely analogous to an affine space.

As a quick-and-dirty rule - not a formal definition - the way you can distinguish between the two concepts is that a word embedding has no natural zero vector (or, slightly more precisely, no word naturally maps to a hypothetical zero vector).

Somewhat more rigorously: an affine space has two conceptually very different classes of vectors: "position" vectors and "displacement" vectors. Only the displacement vectors form a true vector space: you can form arbitrary linear combinations of them. The rules for adding and subtracting position and displacement vectors together are a little more complicated:

  1. The sum or difference of two displacement vectors is a displacement vector (since they form a true vector space).
  2. The difference between two position vectors is a displacement vector. Conversely, a position vector plus a displacement vector equals a position vector.
  3. You can't add two position vectors together.

The affine space itself is the set of all the position vectors (plus the algebraic structure given by the rules for subtracting them).

In a word embedding, the words themselves embed to position vectors. The relations between words correspond to displacement vectors. You can only take arbitrary linear combinations of the relations - i.e. the displacement vectors - and not the words themselves, which you can only subtract to get relations.

  • $\begingroup$ To call it affine space may still be too generous... linear combinations of word vectors may not make sense. The embedding space has not too many dimensions, much less than the number of word senses. This means there are plenty of such linear combos. It's unlikely that they make sense in general. $\endgroup$ Commented Jan 31 at 16:46

My understanding is that "vector embeddings" are more appropriately understood as "metric embeddings".

The principle underlying metric embedding is:

The more a,b are similar in meaning, the smaller the distance d(a,b).

The reason why it occurs can be explained by the following figure:

mapping of nearby points

It is easier for a neural network (which is a continuous function) to map nearby points to the same target, than if there is an exception within the neighborhood (green dot). The latter would require more computing resources, ie, weights. Therefore the above principle holds.

For the example below (left figure), the emergence of seeming "directionality" may be explained by the respective distances of the points being equal (right figure), thus forcing the shape to be a parallelogram. In other words, directionality of vectors may be just a side-effect of the distance metric:

King and Queen example


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.