Why does the MAML split the omniglot data set randomly on every run? I was going through the learn2learn library for meta-learning and saw this:
    classes = list(range(1623))
    random.shuffle(classes)  # todo: wish I wouldn't have copied l2l 
       here and removed this...idk if shuffling this does anything 
       interesting. Doubt it.
    train_dataset: FilteredMetaDataset = 
       l2l.data.FilteredMetaDataset(dataset, labels=classes[:1100])
    validation_dataset: FilteredMetaDataset = 
    l2l.data.FilteredMetaDataset(dataset, labels=classes[1100:1200])
    test_dataset: FilteredMetaDataset = 
    l2l.data.FilteredMetaDataset(dataset, labels=classes[1200:])

doesn't this mean that the data set construction is not deterministic so every experiment result is different? Why is this allowed/ok to do?
see discussion: https://github.com/learnables/learn2learn/issues/361
doesn't this make experiment results across papers incomparable?

cross: https://www.reddit.com/r/pytorch/comments/y3ftqz/is_the_reproducibility_of_omniglot_completely/
 A: You can think of this as, rather than running experiments on a single “dataset”, running experiments on a sample from some meta-distribution (as determined by what your random split of Omniglot characters ended up being).
It’s essentially analogous to, rather than having fixed train/test splits that everyone uses, having everyone choose a different random 80% of the data as their train set.
This does introduce a little bit of noise to the evaluation process; it’s possible that my best run happened to get an “easy” split of the characters, while yours got a “hard” one. But if we do multiple runs and take the average, it’s a well-defined process.
It might help to note that if you wanted to test whether my algorithm’s mean performance is better than yours, using different shuffles of the classes would mean we’d use a normal $t$-test, while if we aligned our shuffles we could use a paired $t$-test.

Note that just removing the shuffle line is likely a bad idea: I haven’t double-checked, but I believe that the Omniglot classes are in a sorted order (by alphabet), so you’d be using a very nonrandom split if you don’t shuffle. If you want reproducibility, as they say in the linked issue, set a random seed.
