I'm developing two k-fold cross-validated models, based on two different data sets, but using the same variables. I plan to then apply both models to each data set and calculate a few model performance measures.

The goals is to purposefully compare how applying the model created from data set 1 to data set 2 is related to a potential decrease in performance against applying the model created from data set 2 to the data in data set 2.

Are there any further steps that must be taken to ensure methodological validity? For example, is there a need to apply the k-fold to a reduced sample and compare the performance of the models applied to the remaining sample data?


1 Answer 1


In a nutshell: yes, you could do this as you suggested.

For training each one of those models, you should do proper inner and outer evaluation within the one dataset you use for training (e.g. repeated k-fold cross validation, using a sufficient amount of partitions and repeats, and the same amount of partitions and repeats for both datasets), then use the obtained results to decide on which model (e.g. different model parametrization) would be best suited for predictions - all using the same, one dataset. Having obtained this final model, you can test it on the other dataset and see for differences in the prediction performance.

Two more things:

  1. If your datasets are small, you will naturally more easily get better/worse results if applying it as held-back test set, because fewer samples cause less stable results. I'd try to use equally sized datasets if possible, and look at both the performance of model 1 on dataset 2 and model 2 on dataset 1.

  2. For the same reason: with e.g. repeated cross validation you will get a number of results for the model you finally choose to test on the other dataset. Those results indicate how well your model usually performs (e.g. average prediction performance), and how wide its performance spread is (e.g. standard deviation, MAD, or quantiles). If you would obtain a rather wide performance spread this indicates, that you will have also have a certain change of just being lucky/unlucky with any held-out test set to obtain a better/worse-than-average prediction performance.

  • 2
    $\begingroup$ Along with these recommendations, I would add one additional stipulation: that the comparisons be done on exactly the same set of k-folds for each dataset. The reason for this is that the theoretical asymptotics of CV don't hold with finite data samples. In other words, the vagaries of random sampling, even with large amounts of data, could invalidate the results of your testing. $\endgroup$ Jul 7, 2016 at 15:23
  • $\begingroup$ @DJohnson True, I updated the answer - thx! $\endgroup$ Jul 7, 2016 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.