37
$\begingroup$

Is there any difference between 'transfer learning' and 'domain adaptation'?

I don't know about context, but my understanding is that we have some dataset 1 and train on it, after which we have another dataset 2 for which we want to adapt our model without retraining from scratch, for which 'transfer learning' and 'domain adaptation' help solve this problem.

According to the field of Convolutional Neural Networks:

  • By 'transfer learning' I mean 'finetuning' [1]

  • In this case [2] it's unsupervised, but should 'domain adaptation' always be unsupervised?

$\endgroup$
1
  • 4
    $\begingroup$ A Survey on Transfer Learning by Pan et. al, 2009 provides some insight. They categorize domain adaptation as a type of transfer learning. $\endgroup$
    – kedarps
    Commented Mar 29, 2017 at 22:53

6 Answers 6

19
$\begingroup$

It seems that there are some disagreement between researchers on what the difference between 'transfer learning' and 'domain adaptation' is.

From {0}:

The notion of domain adaptation is closely related to transfer learning. Transfer learning is a general term that refers to a class of machine learning problems that involve different tasks or domains. In the literature, there isn't yet a standard definition of transfer learning. In some papers it's interchangeable with domain adaptation.

From {1}:

enter image description here


References:

$\endgroup$
12
$\begingroup$

From Hal Daume's article [1]:

The standard classification setting is a input distribution p(X) and a label distribution p(Y|X). Domain adaptation: when p(X) changes between training and test. Transfer learning: when p(Y|X) changes between training and test.

In other words, in DA the input distribution changes but the labels remain the same; in TL, the input distributions stays the same, but the labels change.

  1. https://nlpers.blogspot.com/2007/11/domain-adaptation-vs-transfer-learning.html (mirror)
$\endgroup$
3
  • $\begingroup$ Welcome to Cross Validated! Please take a moment to view our tour. It is preferred that answers are self-contained with proper citations. If you were to use the information from your link to provide a reasonable answer to the question with the link for further reading, this would be a much improved answer. $\endgroup$
    – Tavrock
    Commented Mar 13, 2017 at 19:31
  • 8
    $\begingroup$ I don't believe this explanation. I don't believe HAL here. What if both $p(Y|X)$ and $p(X)$ change at the same time? $\endgroup$
    – caveman
    Commented Apr 27, 2017 at 16:19
  • 3
    $\begingroup$ I agree with @caveman. For many transfer learning scenarios that's the case. $\endgroup$
    – pir
    Commented Sep 9, 2017 at 21:27
3
$\begingroup$

Throughout the literature on transfer learning, there is a number of terminology inconsistencies. Phrases such as transfer learning and domain adaptation are used to refer to similar processes. Domain adaptation is the process of adapting one or more source domains for the means of transferring information to improve the performance of a target learner. The domain adaptation process attempts to alter a source domain in an attempt to bring the distribution of the source closer to that of the target. In the Domain Adaptation setting the source and target domains have different marginal distributions p(X). According to Pan's survey, Transfer Learning is a broader term that can also include the case when there is a difference in the conditional distributions p(Y|X) of the source and target domains. In contrast, Daume discriminates the two terms [1], referring that Domain Adaptation is when p(X) changes between source and target and Transfer learning is when p(Y|X) changes between source and target domains.

  1. https://nlpers.blogspot.com/2007/11/domain-adaptation-vs-transfer-learning.html
$\endgroup$
2
  • $\begingroup$ Would it be possible to add a citation for "Pan's survey"? I think it would make this answer more useful to future readers. $\endgroup$
    – Silverfish
    Commented Jul 16, 2017 at 21:31
  • $\begingroup$ Pan, Sinno Jialin, and Qiang Yang. "A survey on transfer learning." IEEE Transactions on knowledge and data engineering 22, no. 10 (2010): 1345-1359. scholar.google.com/… ; citeseerx.ist.psu.edu/viewdoc/… $\endgroup$ Commented Jul 16, 2017 at 21:33
3
$\begingroup$

I think that "Transfer Learning" is a more general term, and "Domain Adaptation" is a scenario of "Transfer Learning".

[1] Transferable Attention for Domain Adaptation. http://ise.thss.tsinghua.edu.cn/~mlong/doc/transferable-attention-aaai19.pdf

$\endgroup$
0
$\begingroup$

According to [1], domain adaptation is the transfer learning in NLP: "Transfer learning in the NLP domain is sometimes referred to as domain adaptation."

[1] Pan, S. J., and Q. Yang. “A Survey on Transfer Learning.” IEEE Transactions on Knowledge and Data Engineering 22, no. 10 (October 2010): 1345–59. https://doi.org/10.1109/TKDE.2009.191 or https://www.cse.ust.hk/~qyang/Docs/2009/tkde_transfer_learning.pdf

$\endgroup$
0
$\begingroup$

It seems wikipedia has the most concise answer:

Domain adaptation is a subcategory of transfer learning. In domain adaptation, the source and target domains all have the same feature space (but different distributions); in contrast, transfer learning includes cases where the target domain's feature space is different from the source feature space or spaces.

Here is the source.

$\endgroup$
1
  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$ Commented May 12, 2022 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.