4
$\begingroup$

In ML the term "embedding" gets tossed around a lot and the term basically means the construction of a function that takes a high-dimensional vector to a low-dimensional vector in such a way that the high-dimensional vector can be recovered by the low-dimensional vector. (...at least in the case of autoencoders, not sure if word embedding cares about reconstructing the word.)

Embedding was also a word that I encountered in topology and differential geometry. https://en.wikipedia.org/wiki/Embedding

Does there exist any correspondence between the usage of these terminologies between math and ML?

$\endgroup$

1 Answer 1

10
$\begingroup$

In pure mathematics an embedding is any function $f\colon X \to Y$ that is injective and structure-preserving. What do these terms mean?

  • Injective
    Different elements of $X$ are always mapped to different elements of $Y$. Formally: For every $x_1,x_2 \in X$ such that $x_1 \ne x_2$, $f(x_1) \neq f(x_2)$.

  • Structure-preserving
    This depends on the context, but generally means that if some property holds of $x_1, x_2, \ldots, x_n$, then the same property holds of $f(x_1), f(x_2), \ldots, f(x_n)$. For example, if $X$ and $Y$ are equipped with multiplication, then an embedding $f$ would preserve it: For every $x_1,x_2 \in X$, $f(x_1 \cdot x_2) = f(x_1) \cdot f(x_2)$.

The machine-learning use of the term embedding is similar to this. Here we are concerned with (finite) subsets $X \subset \mathbb{R}^n$ and functions $f\colon \mathbb{R}^n \to \mathbb{R}^m$ such that $f(X)$ has approximately the same structure as that of $X$. (Here $f(X)$ denotes the image of $X$ under $f$. Formally: $f(X) = \{f(x) : x \in X \}$.) Like the mathematical definition, exactly what this means depends on the context. Two examples:

  • In the case of t-SNE, a set $X$ of high-dimensional vectors in $\mathbb{R}^n$ ($n>3$) is embedded into low-dimensional space $\mathbb{R}^m$ (typically $m = 2$ or $m = 3$) in such a way that if $x_1$ and $x_2$ are neigbours in $\mathbb{R}^n$, then their images under the embedding are also neighbours in $\mathbb{R}^m$. The embedding is caculated by means of probability densities; full details can be found in the original paper by van der Maaten and Hinton.

  • The idea behind autoencoders is similar: We use an artificial neural network to find an $m$-dimensional approximation of a set $X$ of $n$-dimensional vectors. This amounts to again finding an embedding $f\colon X \to \mathbb{R}^m$ such that if $x_1$ and $x_2$ are neigbours in $\mathbb{R}^n$, then their images $f(x_1)$ and $f(x_2)$ are neighbours in $\mathbb{R}^m$. The embedding $f$ is calculated in a very different way to t-SNE; in fact, an autoencoder finds the embedding $f$ by finding a subsequent embedding $g\colon f(X) \to \mathbb{R}^n$ such that the composed map $g\circ f\colon X \to \mathbb{R}^n$ preserves the identity of each $x \in X$ as closely as possible, i.e. the distance between $x$ and $g\circ f(x)$ should be as small as possible. Another difference with t-SNE is that the embedding generated by the autoencoder should generalise to points outside the training set $X$.

$\endgroup$
1
  • 4
    $\begingroup$ +1 This is a good answer. Just want to mention that embeddings in machine learning aren't restricted to mappings from $\mathbb{n}$ to $\mathbb{m}$. For example, one can embed objects like graphs, strings, etc. Also, we aren't necessarily concerned with finite sets. For example, in the autoencoder case you mentioned, we certainly train on a finite set. But, we'd like the learned mapping to generalize well over the underlying distribution, which is often defined on a continuous space. $\endgroup$
    – user20160
    Jun 20, 2020 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.