I am reading Caruana (1997) Multitask learning (pdf). In the definition of multi task learning, the author states that;
Usually, we do not care how well extra tasks are learned; their sole purpose is to help the main task be learned better.
This is also the goal of transfer learning right? What is the difference then? Are these two terms used interchangeably (I think not)?
Single Task Learning: Giving a set of learning tasks, t1 , t2 , …, t(n), learn each task independently. This is the most commonly used machine learning paradigm in practice.
Multitask Learning: Giving a set of learning tasks, t1 , t2 , …, t(n), co-learn all tasks simultaneously. In other words, the learner optimizes the learning/performance across all of the n tasks through some shared knowledge. This may also be called batch multitask learning. Online multitask learning is more like lifelong learning (see below).
Transfer Learning (or Domain Adaptation): Giving a set of source domains/tasks t1, t2, …, t(n-1) and the target domain/task t(n), the goal is to learn well for t(n) by transferring some shared knowledge from t1, t2, …, t(n-1) to t(n). Although this definition is quite general, almost the entire literature on transfer learning is about supervised transfer learning and the number of source domains is only one (i.e., n=2). It also assumes that there are labeled training data for the source domain and few or no labeled examples in the target domain/task, but there are a large amount of unlabeled data in t(n). Note that the goal of transfer learning is to learn well only for the target task. Learning of the source task(s) is irrelevant.
Lifelong Learning: The learner has performed learning on a sequence of tasks, from t1 to t(n-1). When faced with the nth task, it uses the relevant knowledge gained in the past n-1 tasks to help learning for the nth task. Based on this definition, lifelong learning is similar to the general transfer learning involving multiple source domains or tasks. However, some researchers have a narrower definition of lifelong learning. They regard it as the learning process that aims to learn well on the future task t(n) without seeing any future task data so far. This means that the system should generate some prior knowledge from the past observed tasks to help new/future task learning without observing any information from the future task t(n). The future task learning simply uses the knowledge. This definition makes lifelong learning different from both transfer learning and multitask learning. It is different from transfer learning because transfer learning identifies prior knowledge using the target/future task labeled and unlabeled data. It is different from multitask learning because lifelong learning does not jointly optimize the learning of the other tasks, which multitask learning does.
All content lifted from https://www.cs.uic.edu/~liub/IJCAI15-tutorial.html.
Multi-task Learning and Transfer Learning methods although they have some things in common, they are not the same. Transfer Learning only aims at achieving high performance in the target task by transferring knowledge from the source task, while Multi-task Learning tries to learn the target and the source task simultaneously.
A major assumption in many machine learning and data mining algorithms is that the training and future data must be in the same feature space and have the same distribution. This assumption is weak and in many cases may not hold. For example, imagine that we have a classification task in one domain of interest, but we only have sufficient training data in another domain of interest, where the latter data may be in a different feature space or follow a different data distribution. In Multi-task Learning there is no mandatory to observe a difference in the distributions of the different tasks.
Transfer Learning is a broader topic than Domain Adaptation. The latter is the process that attempts to alter the source domain in a way to bring the distribution of the source closer to that of the target domain. Transfer Learning can include also the case when we want to transfer knowledge from an old task to newer and different one (e.g. in a classification problem that we have different labels), which has some similarities with the old, apart from the above explained situation where we have one task but there is a difference in the distributions of the data (e.g. sample bias etc).
There is a slightly different (and I think better) answer in the ICML 2018 review ( https://towardsdatascience.com/icml-2018-advances-in-transfer-multitask-and-semi-supervised-learning-2a15ef7208ec ), where multi-task learning, domain adaptation and fine-tuning are specific cases of transfer learning.
Lifelong learning (or Metalearning) accumulates knowledge from multiple "historical" tasks seen in the past, with the objective of improving performances on future tasks. It's related to multi-task learning, except that the tasks come from the past, and the targets are in the future.
