5
$\begingroup$

In the context of machine learning, I often hear the term latent space, sometimes qualified with the word "high dimensional" or "low dimensional" latent space.

I am a bit puzzled by this term (as it is almost never defined rigorously).

Can someone please provide a definition or motivation to the concept of a latent space?

$\endgroup$
8
$\begingroup$

Latent space refers to an abstract multi-dimensional space containing feature values that we cannot interpret directly, but which encodes a meaningful internal representation of externally observed events.

Just as we, humans, have an understanding of a broad range of topics and the events belonging to those topics, latent space aims to provide a similar understanding to a computer through a quantitative spatial representation/modeling.

The motivation to learn a latent space (set of hidden topics/ internal representations) over the observed data (set of events) is that large differences in observed space/events could be due to small variations in latent space (for the same topic). Hence, learning a latent space would help the model make better sense of observed data than from observed data itself, which is a very large space to learn from.

Some examples of latent space are:

1) Word Embedding Space - consisting of word vectors where words similar in meaning have vectors that lie close to each other in space (as measured by cosine-similarity or euclidean-distance) and words that are unrelated lie far apart (Tensorflow's Embedding Projector provides a good visualization of word embedding spaces).

2) Image Feature Space - CNNs in the final layers encode higher-level features in the input image that allows it to effectively detect, for example, the presence of a cat in the input image under varying lighting conditions, which is a difficult task in the raw pixel space.

3) Topic Modeling methods such as LDA, PLSA use statistical approaches to obtain a latent set of topics from an observed set of documents and word distribution. (PyLDAvis provides a good visualization of topic models)

4) VAEs & GANs aim to obtain a latent space/distribution that closely approximates the real latent space/distribution of the observed data.

In all the above examples, we quantitatively represent the complex observation space with a (relatively simple) multi-dimensional latent space that approximates the real latent space of the observed data.

The terms "high dimensional" and "low dimensional" help us define how specific or how general the kinds of features we want our latent space to learn and represent. High dimensional latent space is sensitive to more specific features of the input data and can sometimes lead to overfitting when there isn't sufficient training data. Low dimensional latent space aims to capture the most important features/aspects required to learn and represent the input data (a good example is a low-dimensional bottleneck layer in VAEs).

If this answer helped, please don't forget to up-vote it :)

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1 but it does not have to be the case that "we cannot interpret directly" the latent variables, in some cases we can. $\endgroup$ – Tim Dec 27 '19 at 10:17
  • $\begingroup$ Yes true, in some cases we can. However, in most cases, the task is non-trivial because of the challenge in identifying what abstract meaning each dimension could possibly encode. However, there are some well-defined examples with a good analysis of the meanings of these dimensions. Here's a lecture from MIT's 6.S191 course that explains some such analyses - youtu.be/ulLx2iPTIcs $\endgroup$ – Balraj Ashwath Dec 27 '19 at 10:45
  • $\begingroup$ Here you can find example of latent variables that are directly interpretable stats.stackexchange.com/a/430561/35989 $\endgroup$ – Tim Dec 27 '19 at 10:49
1
$\begingroup$

This Quora answer an awesome motivation for latent space. Quote:

The word “latent” means “hidden”. It is pretty much used that way in machine learning — you observe some data which is in the space that you can observe, and you want to map it to a latent space where similar data points are closer together.

For instance, consider these 4 images:

enter image description here

In the pixel space that you observe, there is no immediate similarity between any two images. However, if you were to map it to a latent space, you would want the images on the left to be closer to each other in the latent space than to any of the images on the right. So your latent space captures the structure of your data w.r.t your task.

In LDA, you model the task in a way that documents belonging to similar topics are closer in the latent space of topics.

In word embeddings, you want to map words to a latent vector space such that words with similar meaning are closer in that space.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.