I am working in manufacturing related industry. I am tasked with building some models to predict some physical quantities (for example, size of a hole in the structure) from spectra. The data usually comes in groups, for example my customer would 10 of their products and let us collect data on these products. Since the manufacturing conditions varies from products to products, there's a natural grouping in my data.
Now my question is, when I do cross validation, should I do the train-validation split by groups or not? From sklearn documentation (http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-iterators-for-grouped-data), at first glance it seems to suggest I should since model made from a particular set of products should generalize to other products:
The i.i.d. assumption is broken if the underlying generative process yield groups of dependent samples.
Such a grouping of data is domain specific. An example would be when there is medical data collected from multiple patients, with multiple samples taken from each patient. And such data is likely to be dependent on the individual group. In our example, the patient id for each sample will be its group identifier.
In this case we would like to know if a model trained on a particular set of groups generalizes well to the unseen groups. To measure this, we need to ensure that all the samples in the validation fold come from groups that are not represented at all in the paired training fold.
However, when I think about it, splitting by group seems unnecessary. There is no hidden variable per se that would affect the spectra collected from different groups. In principle, the spectra are only determined by the structure on a product-this seems to be different from the example in the documentation, where supposedly the hidden variable is the patient itself (different individual can be more prone to some disease, or benefits more to some treatments). In fact, blind test on additional data seems to suggest that splitting by groups overestimate the cross validated MSE against splitting without groups.