What is the difference between embedding in pure math and embedding in ML?

Question

In ML the term "embedding" gets tossed around a lot and the term basically means the construction of a function that takes a high-dimensional vector to a low-dimensional vector in such a way that the high-dimensional vector can be recovered by the low-dimensional vector. (...at least in the case of autoencoders, not sure if word embedding cares about reconstructing the word.)

Embedding was also a word that I encountered in topology and differential geometry. https://en.wikipedia.org/wiki/Embedding

Does there exist any correspondence between the usage of these terminologies between math and ML?

Answer 1

In pure mathematics an embedding is any function $f\colon X \to Y$ that is injective and structure-preserving. What do these terms mean?

• Injective
  Different elements of $X$ are always mapped to different elements of $Y$. Formally: For every $x_1,x_2 \in X$ such that $x_1 \ne x_2$, $f(x_1) \neq f(x_2)$.

• Structure-preserving
  This depends on the context, but generally means that if some property holds of $x_1, x_2, \ldots, x_n$, then the same property holds of $f(x_1), f(x_2), \ldots, f(x_n)$. For example, if $X$ and $Y$ are equipped with multiplication, then an embedding $f$ would preserve it: For every $x_1,x_2 \in X$, $f(x_1 \cdot x_2) = f(x_1) \cdot f(x_2)$.

The machine-learning use of the term embedding is similar to this. Here we are concerned with (finite) subsets $X \subset \mathbb{R}^n$ and functions $f\colon \mathbb{R}^n \to \mathbb{R}^m$ such that $f(X)$ has approximately the same structure as that of $X$. (Here $f(X)$ denotes the image of $X$ under $f$. Formally: $f(X) = \{f(x) : x \in X \}$.) Like the mathematical definition, exactly what this means depends on the context. Two examples:

• In the case of t-SNE, a set $X$ of high-dimensional vectors in $\mathbb{R}^n$ ($n>3$) is embedded into low-dimensional space $\mathbb{R}^m$ (typically $m = 2$ or $m = 3$) in such a way that if $x_1$ and $x_2$ are neigbours in $\mathbb{R}^n$, then their images under the embedding are also neighbours in $\mathbb{R}^m$. The embedding is caculated by means of probability densities; full details can be found in the original paper by van der Maaten and Hinton.

• The idea behind autoencoders is similar: We use an artificial neural network to find an $m$-dimensional approximation of a set $X$ of $n$-dimensional vectors. This amounts to again finding an embedding $f\colon X \to \mathbb{R}^m$ such that if $x_1$ and $x_2$ are neigbours in $\mathbb{R}^n$, then their images $f(x_1)$ and $f(x_2)$ are neighbours in $\mathbb{R}^m$. The embedding $f$ is calculated in a very different way to t-SNE; in fact, an autoencoder finds the embedding $f$ by finding a subsequent embedding $g\colon f(X) \to \mathbb{R}^n$ such that the composed map $g\circ f\colon X \to \mathbb{R}^n$ preserves the identity of each $x \in X$ as closely as possible, i.e. the distance between $x$ and $g\circ f(x)$ should be as small as possible. Another difference with t-SNE is that the embedding generated by the autoencoder should generalise to points outside the training set $X$.