6
$\begingroup$

In instance-based transfer learning, it is said that data in the source domain and the target domain are required to be independent and identically distributed. When it says that the data "are required to be independent and identically distributed", I'm assuming it's actually referring to the generative process for the data, since data itself cannot be i.i.d. (that is, i.i.d. is not a property of data – it's a property of random variables), right? If so, then does this mean that the two generative processes are i.i.d. with respect to each other? I don't see how it could be interpreted any other way, but I would just like to confirm.

$\endgroup$
2
  • $\begingroup$ "data in the source domain and the target domain are required to be independent and identically distributed" - link? $\endgroup$ – Aleksejs Fomins Sep 28 '20 at 10:57
  • 1
    $\begingroup$ @AleksejsFomins In Transfer Learning by Yang, Zhang, Dai, and Pan, when discussing relation-based transfer learning approaches, the authors say the following: "Note that, in this category, data in the source domain and the target domain are not required to be independent and identically distributed as the other three categories." Therefore, it is said that instance-based transfer learning requires that data in the source domain and the target domain are required to be independent and identically distributed. $\endgroup$ – The Pointer Sep 28 '20 at 11:08
1
+50
$\begingroup$

I don't know what exactly was meant by the original statement, but it may include some or all of below statements

  1. Source data generative process is i.i.d
  2. Target data generative process is i.i.d
  3. The processes are i.i.d with each other

All of these are very sensible standard assumptions, because if this is not the case, one can design adversarial generative processes for which the method will work very differently than for i.i.d datasets.

For points 1. and 2. a bad example is all data being the same. For 3., imagine if source and target are forced to explore very different parts of phase space. This requires them not being i.i.d, but will result in transfer learning being useless, as there will be no overlap

EDIT: Some clarifications based on comments

Q1) If you cannot assume i.i.d, it means that you have to, in principle, be ready to deal with any non-iid datasets. Adversarial means that a bad guy can come and, out of all possible generating processes that are allowed by your assumptions, select the one that screws you up the most.

Q2) Phase-space is the multidimensional space spanned by all variables of the system. For example, if you input a 10x10 pixel colored image, your phase-space will have 10x10x3 = 300 dimensions. Any knowledge can be represented as a partition of the phase space. For example, all possible 10x10 colored images of a cat will take a certain volume in phase space. While this volume need not be convex, it is typically concentrated in some part of the phase space, if your object (a cat, that is) is well-defined. I highly recommend taking introductory courses on dynamical systems and information theory before attempting to study advanced topics such as transfer learning. I think it is beneficial to think generally about what knowledge means, how it is represented and related to other knowledge before going into details of implementation of specific knowledge-processing devices

$\endgroup$
4
  • $\begingroup$ Yes, your three point summary is precisely how I was thinking about this! But, due to my inexperience, I wasn't sure if what I was thinking about was valid or nonsense. $\endgroup$ – The Pointer Sep 28 '20 at 11:23
  • $\begingroup$ 1. "All of these are very sensible standard assumptions, because if this is not the case, one can design adversarial generative processes for which the method will work very differently than for i.i.d datasets." Would you please explain what you mean by this statement? 2. What is meant by "phase space"? This papers.nips.cc/paper/943-phase-space-learning.pdf paper describes it as "The phase space is the space of the dependent variables ($X$)", but I don't really understand what this means. $\endgroup$ – The Pointer Sep 28 '20 at 11:23
  • $\begingroup$ 3. If the source data generative process is i.i.d, and the target data generative process is i.i.d, then that does not necessarily mean that the processes are i.i.d with each other, right? $\endgroup$ – The Pointer Sep 28 '20 at 11:26
  • 1
    $\begingroup$ For your last point, yes, for sure not. The probability that it rains in Scotland on any given day is an i.i.d process (trust me on that one), and the probability that a random person gets wet on the street in Scotland is also i.i.d, but the two processes are very much related $\endgroup$ – Aleksejs Fomins Sep 28 '20 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.