Do word vectors obtained via word embedding techniques really form a vector space?

Question

Word embedding refers to feature learning techniques in natural language processing where words are mapped to vectors of real numbers in a low-dimensional space, the embedding space.

Similar to other feature learning (representation learning) techniques, the word vectors are learned within a neural network classifier (e.g. the weights a hidden layer) or by using dimensionality reduction methods (e.g. LSA / PCA) etc. The collection of learned word vectors is basically a matrix, where each row is the learned representation of a word, and each column corresponds to a learned latent feature.

Now, the literature sometimes refers to the embedding space as to a vector space. And I've been recently challenged about the correctness of using the term vector space in this context.

---

Is the usage of the term vector space acceptable e.g. in machine learning and math communities?

Do word vectors obtained via word embedding techniques really form a vector space?

Is it even practical to concern about this?

Answer 1

A few (weak) arguments against using the term "vector space"

The commutativity property of vector addition does not always hold in semantics. Therefore, this property shouldn't (always) hold in the embedding space either. Thus, the embedding space should not be called a vector space.

E.g. attempt to treat semantic composition as vector addition in the vector space:

$v_{rescue \ dog} = v_{rescue} + v_{dog}$ (a dog which is trained to rescue people)

$v_{dog \ rescue} = v_{dog} + v_{rescue}$ (the operation of saving a dog)

The phrases "rescue dog" and "dog rescue" mean different things, but in our hypothetical vector space, they would (incorrectly) have the same vector representation (due to commutativity).

Similarly for the associativity property.

Regarding the existence of additive inverse - this property is often violated in semantics, as many words are non-opposable (e.g. bee). Therefore, one might say it is wrong if the embedding space satisfies the condition of additive inverse. Thus, the embedding space shouldn't be called a vector space.

And so on.

---

A few (common-sense) arguments in favour of using the term "vector space"

A misuse of the algebraical term vector space does no harm in this context.

Choosing to model the learned word features and word relationships in a vector space implies, indeed, erroneous assumptions and constraints which are irrelevant to natural language, but this can't be coined as unacceptable. This issue seems related to choosing a model with a high bias, which can't capture well all the particularities of the real-world data (natural language).

Furthermore, the term vector space model has never seemed inappropriate. And word vectors obtained through word embedding techniques are, actually, part of this family of models.