Why is Permuted MNIST good for evaluating continual learning models?

Question

While I was reading papers about continual learning, I found that many researchers use permutated MNIST to evaluate their approach.

I understand what it is but it is not clear to me why they use it?

What I understand is that they were trying to introduce noise (by applying a random permutation to the image) but the permutated images are very noisy which cannot be recognised even by a human.

Applying blur, rotation, or some distortion are understandable but why permuting the pixel?

PS. an example of the paper I mentioned: Three scenarios for continual learning

Answer 1

Actually as stated in this paper https://arxiv.org/abs/1805.09733 the Permuted MNIST evaluation is not a good method to measure Continual Learning performance. This is because the permuted images differ too much from the original, which is not a good representation of real-world scenarios. Normally a new task with a new dataset could have similar images which could lead to the network making false positive predictions.

Answer 2

Since the authors use an MLP which does not exploit the spatial relationship between the pixels as a CNN might, it is no more easy or difficult for their model to learn MNIST versus permuted MNIST.

the permutated images are very noisy which cannot be recognised even by a human.

While the images look noisy, the amount of actual noise is close to 0, since again, they are just permuted versions of noise-free MNIST.

Applying blur, rotation, or some distortion are understandable but why permuting the pixel?

Blur, rotation, or distortion would actually not work as well in this setting, since that would test the ability of the model to generalize to blur/rotation/distortion, whereas the actual goal is to test continual learning.

Answer 3

I found a paper that could clearly answer my own question.

https://arxiv.org/abs/1708.02072

Data Permutation Experiment - The elements of every feature vector are randomly permuted, with the permutation held constant within a session, but varying across sessions. The model is evaluated on its ability to recall data learned in prior study sessions. Each session contains the same number of examples.

Answer 4

Within Continual Learning, there are three main problem paradigms:

1. Task-Incremental Learning (where we want the model to solve multiple distinct tasks)
2. Class-Incremental Learning (where we want the model to solve a classification problem, while being presented with additional classes in each new task)
3. Domain-Incremental Learning (where we want the model to answer the same question, but the underlying distribution of the data changes with each task).

Permuted MNIST¹ is designed to present an example of the latter problem, where the question (and output labels) are the same:

"Is this an image of a 0, 1, 2, 3, ... or 9?"

but the underlying distribution of the input data differs with each task.

Note however that Permuted NIST has been criticised² for not providing a realistic example of an evaluation setting for a Continual Learning model (i.e. there is little to no similarity between the tasks). A more natural example may be, you have a model designed to classify images of cats and dogs based on headshot photos taken face on indoors. You receive new data for nominally the same task (classify pictures of cats and dogs), but the images are from a different distribution where a subset of learnable features are hypothesized to be shared (e.g. colour of animals, facial features etc), e.g:

• photos of the animals in profile
• photos of different species of cat and dog
• photos in a different context (park vs indoors)

---

1. Permuted MNIST was originally proposed in An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks (2015), and has since been adopted as a common test of Domain-IL performance in Continual Learning (e.g. Three scenarios for continual learning):

   To test this kind of learning problem, we designed a simple pair of tasks, where the tasks are the same, but with different ways of formatting the input. Specifically, we used MNIST classification, but with a different permutation of the pixels for the old task and the new task. Both tasks thus benefit from having concepts like penstroke detectors, or the concept of penstrokes being combined to form digits. However, the meaning of any individual pixel is different. The net must learn to associate new collections of pixels to penstrokes, without significantly disrupting the old higher level concepts, or erasing the old connections between pixels and penstrokes.

2. Towards Robust Evaluations of Continual Learning (2019):

   Cross-task resemblances Input data from later tasks must resemble old tasks enough that they at least sometimes result in confident predictions of old classes, early in training. The widely used Permuted MNIST (see §4.2.1) which violates this...