Comparing performance of two k-fold cross-validated models

Question

I'm developing two k-fold cross-validated models, based on two different data sets, but using the same variables. I plan to then apply both models to each data set and calculate a few model performance measures.

The goals is to purposefully compare how applying the model created from data set 1 to data set 2 is related to a potential decrease in performance against applying the model created from data set 2 to the data in data set 2.

Are there any further steps that must be taken to ensure methodological validity? For example, is there a need to apply the k-fold to a reduced sample and compare the performance of the models applied to the remaining sample data?

Answer 1

In a nutshell: yes, you could do this as you suggested.

For training each one of those models, you should do proper inner and outer evaluation within the one dataset you use for training (e.g. repeated k-fold cross validation, using a sufficient amount of partitions and repeats, and the same amount of partitions and repeats for both datasets), then use the obtained results to decide on which model (e.g. different model parametrization) would be best suited for predictions - all using the same, one dataset. Having obtained this final model, you can test it on the other dataset and see for differences in the prediction performance.

Two more things:

1. If your datasets are small, you will naturally more easily get better/worse results if applying it as held-back test set, because fewer samples cause less stable results. I'd try to use equally sized datasets if possible, and look at both the performance of model 1 on dataset 2 and model 2 on dataset 1.

2. For the same reason: with e.g. repeated cross validation you will get a number of results for the model you finally choose to test on the other dataset. Those results indicate how well your model usually performs (e.g. average prediction performance), and how wide its performance spread is (e.g. standard deviation, MAD, or quantiles). If you would obtain a rather wide performance spread this indicates, that you will have also have a certain change of just being lucky/unlucky with any held-out test set to obtain a better/worse-than-average prediction performance.