Assuming we want to learn k tasks jointly, and the data for all tasks are available. We may either train a model with parallel multi-task learning (eg. each batch is a mixture of samples from the k tasks), or present tasks sequentially (eg. switch to a different task once every 5k time steps). The latter is kind of like continual learning, except that the set of tasks is fixed and there won't be new ones. Which training paradigm yields better results? Any paper that gives theoretical analysis or makes empirical comparisons?
The answer surely depends on the learning algorithm. If using typical optimization algorithms (e.g. some variant of stochastic gradient descent, since we're talking about online learning), I would choose to train on multiple tasks simultaneously when possible. When training on one task at a time, the learning algorithm may adjust parameters in a way that increases performance on the current task, but hurts performance on previously learned tasks. In extreme cases, performance on previous tasks could fall all the way back to baseline. This is a well known problem called catastrophic forgetting. Mitigating this problem (e.g. using alternative learning algorithms) is an active research topic.