Learning from multiple very varied data sets?

Question

Suppose we have a set of objects $X$ (e.g. individual humans). Suppose also that humans can be described by a set of (potentially very high-dimensional) variables $V_i$, (e.g. $V_1$ is a picture of their face, $V_2$ is their name, $V_3$ is a picture of their house, etc).

Now suppose we have multiple datasets $(D_i)$ all of which capture very different information about $X$, i.e.:

• the $D_i$ have very different types. e.g. $D_1$ contains images, while $D_2$ contains audio

• the $D_i$ have different levels of aggregation. e.g. perhaps $D_1$ gives pictures of individuals, and their location, while $D_2$ gives wealth, but only contains information on the average wealth at the city level.

• The $D_i$ vary in different ways. e.g. all elements of $D_1$ have individuals from New York, but vary in terms of income level, while all elements of $D_2$ are from various different cities but are all upper middle class.

• perhaps more...?

I want to learn more about learning problems like this. What keywords/resources should I look for?

• Is there a name for such "very varied datasets about the same underlying objects"? (or a name for the problem containing such datasets?) I'd like to look this up and read more about it. I guess it has to do with "big data" but I'm hoping there are more specific keywords I could search for.

• Are there names for algorithms that work for these problems? (or other keywords that can help me look for algorithms)

• Is there theory about this type of problem? e.g. in statistical learning theory.

Answer 1

I think this falls under the category of data fusion. The specific methods include, but are not limited to multi-kernel learning.

To deal with data on different levels (descriptors for people, descriptors for cities) one would usually use hierarchical models or random effects/mixed effects models (same thing).

Answer 2

This is very broad problem, so there's probably a variety of approaches by different communities.

A version of this is called multiview learning, where the data might have different modalities. So view $V_1$ could be the text associated with a webpage, and $V_2$ could be the metadata associated with the webpage. One of the earlier ML examples of doing multi-view learning was co-training developed by Blum and Mitchell. This is a form of semi-supervised learning where you can alternate between the two view to extend partial labels to the whole dataset. Blum and Mitchell, also have some discussion of this from a PAC learning PoV. You can find a survey of multiview learning here.