This paper Deep Visual-Semantic Alignments for Generating Image Descriptions on image captioning proposed a Multimodal Recurrent Neural Network architecture.

From my understanding, the multimodal RNN is essentially a language model conditioned on a given image vector.

Somehow I'm confused what "multimodal" refers to in that paper, how do I interpret this? Does it mean that since it is conditioned on an image vector, the language model becomes multimodal?


Multimodal literally means "multiple modes", where vision, language, audio, lidar, etc data are each a different mode of information provided by the world. So you can call any model which deals with more than one mode of sensory input a multimodal model.

| cite | improve this answer | |
  • $\begingroup$ Thanks, I mean I basically understand the concept of a multimodal distribution, in that paper the multimodal RNN model outputs a categorical probability every step (just like many other RNN models), so I'm confused how the name multimodal should be understood in that context. $\endgroup$ – dontloo Feb 12 '19 at 6:39
  • 1
    $\begingroup$ @dontloo multimodal is an overloaded term. the other sense of the word is a distribution with multiple peaks. that is not what is meant in this paper. $\endgroup$ – shimao Feb 12 '19 at 6:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.