Non-uniform data: how to build a prediction model with high performance on unseen groups of data? How to deal with non-uniform data?
There are groups in my data. Samples in the same group share the majority of feature values.
What approach to model training can I use to guarantee high performance on unseen groups?
Because predicting new samples from the same groups used for training (i.e. splitting samples from the same group between training, validation and test sets) is much easier, I do not split samples from one group between the sets. Instead, I split groups: some groups go into the training set, some — to the test set, and some — to the validation set.
However, there is a problem with this approach: model performance largely depends on the selection of the training groups. Some data groups are similar to others, so predicting groups that are similar to the ones used in training is easy. On the other hand, predicting a group (from a test or validation set) that is quite different from the groups used in training is hard.
The number of groups is not large (about 10).
The "solution" I have in mind is to train one model on each possible selection of test and validation groups while selecting one group for the test set and one group for the validation set. The mean model prediction score on validation sets of all selections would be the assessment of the training parameters. To evaluate another set of training parameters, I would need to train new models on all possible selections of the test/validation sets again.
This approach requires a lot of calculations because the number of selections is large: for 10 groups, it is 10 variants of selecting a group for the test set * 9 variants of selecting a group for the validation set = 90 selections. That means I have to train 90 models to evaluate just one set of training parameters.
Another problem is that I use the validation set for selecting optimal training parameters.
Yet another problem is that I use Neural Networks as prediction models, and training a Neural Network is itself unpredictable: using the same training samples and parameters in two training sessions may result in quite different NN performance.
What would be a good approach to building a NN that provides high prediction accuracy on yet unseen groups, an approach that would not require too much time and computational resources?
 A: In general, if you fit a model on data from one group (population), you cannot expect it to work well on data from another group/population.
I presume that, while the groups are all different, you expect them to be similar to a certain degree, such that models trained on one group are not completely off with respect to another of your groups. And in the same way, completely new groups are still somehow similar to your ten training groups. If that wasn't the case, i.e. if your groups were really completely unrelated, there would be no point in learning models.
In other words, I presume that your ten groups are samples from a density over the set of datasets that doesn't have too large a variance.
In this situation, I would learn a neural network on the combined data of groups. Split your ten groups into one group for testing and the other nine groups to do cross-validation on. Choose e.g. each of those nine groups once as validation data and train on the combination of the remaining eight groups.
If you have enough data this should work. Just make sure that you properly regularize your NN: if you don't, they might overfit and then only work well on the training groups.
As always with cross-validation, don't forget to, in the end, fit a final model to all of your nine non-test groups. This final model then is tested against the hold-out test group.
