I don't have computer vision background, yet when I read some image processing and convolutional neural networks related articles and papers, I constantly face the term, multimodal embedding. Can anybody provide some intuition behind this term?

I usually encounter this term in image captioning and image recognition.


2 Answers 2


Follow the link to its pdf for some multimodal embeddings. Multimodal refers to an admixture of media, e.g., a picture of a banana with text that says "This is a banana." Embedding means what it always does in math, something inside something else. A figure consisting of an embedded picture of a banana with an embedded caption that reads "This is a banana." is a multimodal embedding.

Edit For @Herbert From this: In the context of neural networks, embeddings are low-dimensional, learned continuous vector representations of discrete variables. Elsewhere, one finds this: An embedding is a relatively low-dimensional space into which you can translate high-dimensional vectors. Embeddings make it easier to do machine learning on large inputs like sparse vectors representing words. Ideally, an embedding captures some of the semantics of the input by placing semantically similar inputs close together in the embedding space. An embedding can be learned and reused across models.

In terms of what embedding usually means, the neural network definition of embedding seems to me to be particular to that field. That is, it has some of the characteristics features of an embedding in a larger sense, but is sometimes more figurative and sometimes more exact.

In general, the word embedded can be used somewhat figuratively or "metaphorically." For example, a dictionary definition is

verb (used with object), em·bed·ded, em·bed·ding.

  1. to fix into a surrounding mass: to embed stones in cement.
  2. to surround tightly or firmly; envelop or enclose: Thick cotton padding embedded the precious vase in its box.
  3. to incorporate or contain as an essential part or characteristic: A love of color is embedded in all of her paintings.

I am not a grammarian, but it seems to me that the third definition above is a figure of speech, a hyperbole, a metaphor, and is inexact. Whereas such things are common linguistically, they are not literal, and in that sense, the usage of the word embedded for neural networks is somewhat jargonesque.

  • 2
    $\begingroup$ Thanks, Carl! That link really helped me to get through those papers. $\endgroup$
    – kanik garg
    Commented Dec 17, 2017 at 12:32
  • $\begingroup$ In the context for neural networks, I always assumed embedding means "the vector representation of", like word embeddings and word vectors would be the same. In this particular case, the embedding or vector would represent the text and the image together, and ideally would also capture caption-image relations. Is this incorrect? $\endgroup$
    – Herbert
    Commented Dec 18, 2020 at 10:22
  • $\begingroup$ @Herbert Well, it's a bit complicated. I put a response into the answer, as it was too long for a comment. $\endgroup$
    – Carl
    Commented Dec 18, 2020 at 12:50
  • $\begingroup$ @Herbert Yes, all of that is correct. I think this answer is unnecessarily overcomplicating the matter. It's just projections of data points based on their surrounding context (=embedding) that is most likely to occur. $\endgroup$
    – runDOSrun
    Commented Apr 23, 2021 at 7:17

The term "modal" in this context refers to data type. For example, text is a data type. Images is another.

Embedding refers to a mapping of an observation to a point in an N-dimensional space. For example, the mean (R, G, B) value of an image is a mapping of that image to a point in a 3-dimensional space. For another example, the state-of-the-art Sentence BERT (Bidirectional Encoder Representation from Transformers) language model can map a sentence to a 768-dimensional space such that similar sentences are closer to each other in that hyperspace.

Thus, multimodal embedding refers to a mapping of an observation that contains different data types to a single point in an N-dimensional space. For example, mapping of a news article containing both text and images to a point in space.

You're encountering the term multimodal embedding in image captioning because it involves both images and text.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.