# Learning from multiple very varied data sets?

Suppose we have a set of objects $$X$$ (e.g. individual humans). Suppose also that humans can be described by a set of (potentially very high-dimensional) variables $$V_i$$, (e.g. $$V_1$$ is a picture of their face, $$V_2$$ is their name, $$V_3$$ is a picture of their house, etc).

Now suppose we have multiple datasets $$(D_i)$$ all of which capture very different information about $$X$$, i.e.:

• the $$D_i$$ have very different types. e.g. $$D_1$$ contains images, while $$D_2$$ contains audio

• the $$D_i$$ have different levels of aggregation. e.g. perhaps $$D_1$$ gives pictures of individuals, and their location, while $$D_2$$ gives wealth, but only contains information on the average wealth at the city level.

• The $$D_i$$ vary in different ways. e.g. all elements of $$D_1$$ have individuals from New York, but vary in terms of income level, while all elements of $$D_2$$ are from various different cities but are all upper middle class.

• perhaps more...?

• Is there a name for such "very varied datasets about the same underlying objects"? (or a name for the problem containing such datasets?) I'd like to look this up and read more about it. I guess it has to do with "big data" but I'm hoping there are more specific keywords I could search for.

• Are there names for algorithms that work for these problems? (or other keywords that can help me look for algorithms)

• Could you also talk about the learning objective? You only described the data available, what would the goal? – Konstantin Nov 12 '19 at 11:40
• @Konstantin, I kind of wanted to leave that open, because one could imagine various goals, and I'm interested in the whole topic. But e.g. imagine we want to generalize some property $f:X\to Y$ from our datasets, i.e. find $f(x)$ for $x$ not in our dataset (assuming we don't know $f$ beforehand). – user56834 Nov 12 '19 at 15:13

A version of this is called multiview learning, where the data might have different modalities. So view $$V_1$$ could be the text associated with a webpage, and $$V_2$$ could be the metadata associated with the webpage. One of the earlier ML examples of doing multi-view learning was co-training developed by Blum and Mitchell. This is a form of semi-supervised learning where you can alternate between the two view to extend partial labels to the whole dataset. Blum and Mitchell, also have some discussion of this from a PAC learning PoV. You can find a survey of multiview learning here.