Confusion about multimodal machine learning

Question

I recently browsed through this tutorial on multimodal data.

Attention: Multimodal in the sense of feature of very different type, that express the same thing

-think picture and voice of someone talking

-not in the sense of a probability distribution with multiple modes, according to the slides.

What I do not understand about the whole approach to multimodal machine learning techniques is why they are only applied only in the case of features that obviously express the same thing - and not also in the case of features for which it is not clear if they express the same thing.

Example: you want to predict which webshop visitors are likely to buy stuff and you measure, e.g., their mouse movements as well as the time they spent looking at products; these features might be correlated too. For example, the whole search for correlations between the feature that are assumed to express the same thing could also be applied in the case I outlined above. Of course, applying multimodal techniques in the latter case might not yield anything, but it seems no one is even trying.

Answer 1

The basic idea behind multimodal learning techniques and especially those built on top of deep learning is that the different data modalities offer complementary information and one can profit from jointly modeling them. That is compared to using only a single modality you can obtain better results. The different modalities should be adequately different because this motivates the types of architectures shown in the tutorial. Also, the deep learning machinery offer flexible ways (convolutions or sequence models, ..) and extensible architectures to treat the modalities separately and then combine them which is promising.

Your example, which is a valid one, is a more traditional approach where you perform feature extraction on a single modality (behavioral features in your case) and then feed a classifier. Had you, however, access to a different informative modality, e.g., the sounds surrounding the person who browses assuming they do provide some information from your task, you could try multimodal learning. People have been trying to take advantage of that and is also among the straight-forward baselines against a multimodal system should compare. Boosting for instance builds on similar ideas (it's not the same explicitly but just to make the point) where you iteratively train on different features and data subsets to combine weak learning into better performing learners.