Parallel multi-task learning vs. continual learning Assuming we want to learn k tasks jointly, and the data for all tasks are available. We may either train a model with parallel multi-task learning (eg. each batch is a mixture of samples from the k tasks), or present tasks sequentially (eg. switch to a different task once every 5k time steps). The latter is kind of like continual learning, except that the set of tasks is fixed and there won't be new ones. Which training paradigm yields better results? Any paper that gives theoretical analysis or makes empirical comparisons?
 A: The answer surely depends on the learning algorithm. If using typical optimization algorithms (e.g. some variant of stochastic gradient descent, since we're talking about online learning), I would choose to train on multiple tasks simultaneously when possible. When training on one task at a time, the learning algorithm may adjust parameters in a way that increases performance on the current task, but hurts performance on previously learned tasks. In extreme cases, performance on previous tasks could fall all the way back to baseline. This is a well known problem called catastrophic forgetting. Mitigating this problem (e.g. using alternative learning algorithms) is an active research topic.
A: 
... and the data for all tasks are available.

Continual learning (CL) is usually framed under the assumption that training data for previously seen tasks is not available for training on the current task. Under this assumption, "parallel multi-task training" (or joint-training as it is usually termed in CL literature) is presented as a sensible upper bound for performance of continual learning architectures:

We included the following two baselines:

*

*None: The model was sequentially trained on all tasks in the standard way. This is also called fine-tuning, and can be seen as a lower bound.

*Offline: The model was always trained using the data of all tasks so far. This is also called joint training, and was included as it can be seen as an upper bound.



*

*Three scenarios for continual learning (2019)

As shown in their results, the joint-training method does in this instance provide better performance to all models tested (table 4, table 5).
Note: the assumption that this will provide an upper bound, while reasonable and empirically justified for current CL architectures, is not rigorously defended. It is imaginable that for sufficiently similar tasks, for a specific ordering of tasks (cf curriculum learning), performance on previously seen tasks' test data may increase under a continual learning framework compared to parallel joint training:

Joint Training is often used as an upper bound performance for CL. However, in the presence of a strong positive forward and backward transfer this may not be true.


*

*Continual Learning for RNNs: An Empirical Evaluation
