Transfer Learning: data in the source domain and the target domain are required to be independent and identically distributed

Question

In instance-based transfer learning, it is said that data in the source domain and the target domain are required to be independent and identically distributed. When it says that the data "are required to be independent and identically distributed", I'm assuming it's actually referring to the generative process for the data, since data itself cannot be i.i.d. (that is, i.i.d. is not a property of data – it's a property of random variables), right? If so, then does this mean that the two generative processes are i.i.d. with respect to each other? I don't see how it could be interpreted any other way, but I would just like to confirm.

Answer 1

I don't know what exactly was meant by the original statement, but it may include some or all of below statements

1. Source data generative process is i.i.d
2. Target data generative process is i.i.d
3. The processes are i.i.d with each other

All of these are very sensible standard assumptions, because if this is not the case, one can design adversarial generative processes for which the method will work very differently than for i.i.d datasets.

For points 1. and 2. a bad example is all data being the same. For 3., imagine if source and target are forced to explore very different parts of phase space. This requires them not being i.i.d, but will result in transfer learning being useless, as there will be no overlap

EDIT: Some clarifications based on comments

Q1) If you cannot assume i.i.d, it means that you have to, in principle, be ready to deal with any non-iid datasets. Adversarial means that a bad guy can come and, out of all possible generating processes that are allowed by your assumptions, select the one that screws you up the most.

Q2) Phase-space is the multidimensional space spanned by all variables of the system. For example, if you input a 10x10 pixel colored image, your phase-space will have 10x10x3 = 300 dimensions. Any knowledge can be represented as a partition of the phase space. For example, all possible 10x10 colored images of a cat will take a certain volume in phase space. While this volume need not be convex, it is typically concentrated in some part of the phase space, if your object (a cat, that is) is well-defined. I highly recommend taking introductory courses on dynamical systems and information theory before attempting to study advanced topics such as transfer learning. I think it is beneficial to think generally about what knowledge means, how it is represented and related to other knowledge before going into details of implementation of specific knowledge-processing devices