# Splitting data to study datsets based on in- and out-of-distribution

I found an interesting paper for which they evaluate the in and outdistribution on different splittings of the dataset. For this they propose four different in distribution settings. The two I'm interested are Train-to-train and Mxed-to-test, citing:

Train-to-train (train on P_train, evaluate on P_train). In the train-to-train setting, we train a model on Dtrain and evaluate on a separate but identically-distributed test set D_train heldout drawn from P_train.

Mixed-to-test (train on a mixture of P_train and P_test, test on P_test). In the mixed-to-test setting, we train a model on a mixture of data from P_train and P_test and then evaluate it only on P_test.

The way I would approach this for train-to-train is to split I would randomly split the data (to receive iid) of D_train and D_train_heldout. The thing I'm unsure about for both train-to-train and mixed-to-train of is how to go make sure that my data come from a specific distribution. This specifically concerns the question of taking a mixture of P_train and P_test in mixed-to-test. One vanilla case I could think of is to collect a dataset (which would be assumed to be P_train) and taking a second dataset under different conditions. For train-to-train I would train it on the first dataset while splitting randomly in train and evluation. For mixed-to-test I would train it both on first and second dataset and evaluate it on the second.

The paper is linked here with the cited part being on page 18.

Does anybody have any idea of how to tackle this problem?

Edit:

What they are trying to study how the dataset behave with in- and out distributional data. What you normally do when training your data is to randomly split into iid data (last case within the paper on page 19). The question now is understand how the dataset behaves on out-distribution data. For this they proposed different methods of splitting the data. Since I want to apply this by myself I want to get a better understanding of how to approach this (maybe by giving me a more practical insight). This specifically concerns Mixed-to-test where to train on a mixture of P train and P test.

• I'm afraid we don't have access to your hard drive so won't be able to see the paper. Could you provide a link to an online resource? It is not exactly clear what the problem is here. Why not just have a held-out test set for testing? Why would you first split the data to train and test, then mix it and split it again? It seems like the description is missing some important details.
– Tim
Jun 17 at 8:39
• Is it this paper arxiv.org/abs/2012.07421 ?
– Tim
Jun 17 at 8:43
• Haha sorry. That's the paper. Jun 17 at 9:14