What is a latent space?

Question

In the context of machine learning, I often hear the term latent space, sometimes qualified with the word "high dimensional" or "low dimensional" latent space.

I am a bit puzzled by this term (as it is almost never defined rigorously).

Can someone please provide a definition or motivation to the concept of a latent space?

Answer 1

Latent space refers to an abstract multi-dimensional space containing feature values that we cannot interpret directly, but which encodes a meaningful internal representation of externally observed events.

Just as we, humans, have an understanding of a broad range of topics and the events belonging to those topics, latent space aims to provide a similar understanding to a computer through a quantitative spatial representation/modeling.

The motivation to learn a latent space (set of hidden topics/ internal representations) over the observed data (set of events) is that large differences in observed space/events could be due to small variations in latent space (for the same topic). Hence, learning a latent space would help the model make better sense of observed data than from observed data itself, which is a very large space to learn from.

Some examples of latent space are:

1) Word Embedding Space - consisting of word vectors where words similar in meaning have vectors that lie close to each other in space (as measured by cosine-similarity or euclidean-distance) and words that are unrelated lie far apart (Tensorflow's Embedding Projector provides a good visualization of word embedding spaces).

2) Image Feature Space - CNNs in the final layers encode higher-level features in the input image that allows it to effectively detect, for example, the presence of a cat in the input image under varying lighting conditions, which is a difficult task in the raw pixel space.

3) Topic Modeling methods such as LDA, PLSA use statistical approaches to obtain a latent set of topics from an observed set of documents and word distribution. (PyLDAvis provides a good visualization of topic models)

4) VAEs & GANs aim to obtain a latent space/distribution that closely approximates the real latent space/distribution of the observed data.

In all the above examples, we quantitatively represent the complex observation space with a (relatively simple) multi-dimensional latent space that approximates the real latent space of the observed data.

The terms "high dimensional" and "low dimensional" help us define how specific or how general the kinds of features we want our latent space to learn and represent. High dimensional latent space is sensitive to more specific features of the input data and can sometimes lead to overfitting when there isn't sufficient training data. Low dimensional latent space aims to capture the most important features/aspects required to learn and represent the input data (a good example is a low-dimensional bottleneck layer in VAEs).

If this answer helped, please don't forget to up-vote it :)

Answer 2

Latent space is a vector space spanned by the latent variables. Latent variables are variables which are not directly observable, but which are $-$ up to the level of noise $-$ sufficient to describe the data. I.e. the observable variables can be derived (computed) from the latent ones.

Let me use this image, adapted from GeeksforGeeks, to visualise the idea:

Each observable data point has four visible features: the $x, y,$ and $z$-coordinates, and the colour. However, each point is uniquely determined by a single latent variable, $\varphi$ (phi in the python code).

phi = np.linspace(0, 1, 100)       # the latent variable
x = phi * np.sin(25 * phi)         # 1st observable: x-coordinate
y = np.exp(phi) * np.cos(25 * phi) # 2nd observable: y-coordinate
z = np.sqrt(phi)                   # 3rd observable: z-coordinate
c = x + y                          # 4th observable: colour

This is, of course, just a toy example. In practice, you often have many, maybe even millions of observable variables (think of pixel values in images), but they can be sufficiently well computed from a much smaller set of latent variables. In such cases it may be useful to perform some kind of dimensionality reduction.

As a real-world example, consider spectra of light-emitting objects, like stars. A spectrum is a long vector of values, light intensities at many different wavelengths. Modern spectrometers measure the intensity at thousands of wavelengths. However, each spectrum can be quite well described by the star's temperature (through the black body radiation law) and the concentration of different elements (for the absorption lines). These are likely to be way less then thousands, maybe only a dozen or two. That would be a low dimensional latent space.

Note, however, that it's not necessary for the latent space to be smaller than the observable space. It is completely conceivable for many latent variables to influence few observable ones. For example, the value of a particular share at the stock market at a certain point in time is a single value, but it is likely due to many influences which are mostly unknown.

In machine learning I've seen people using high dimensional latent space to denote a feature space induced by some non-linear data transformation which increases the dimensionality of the data. The idea (or the hope) is to achieve linear separability (for classification) or linearity (for regression) of the transformed data. For example, support vector machines use the kernel trick to transform the data, but the transformation is only implicit, given by the kernel function. Such data are "latent" in the sense that you (or the algorithm) never know their values; you only know the dot products of pairs of points.

Answer 3

This article would give you a great understanding about latent space,as a short review :

The latent space is simply a representation of compressed data in which similar data points are closer together in space.

Latent space is useful for learning data features and for finding simpler representations of data for analysis.

We can understand patterns or structural similarities between data points by analyzing data in the latent space, be it through manifolds, clustering, etc.

We can interpolate data in the latent space, and use our model’s decoder to ‘generate’ data samples.

We can visualize the latent

space using algorithms such as t-SNE and LLE, which takes our latent space representation and transforms it into 2D or 3D.

you can also see a great description in here and here