Group KFold for regression problems: When is it helpful? I am working in manufacturing related industry. I am tasked with building some models to predict some physical quantities (for example, size of a hole in the structure) from spectra. The data usually comes in groups, for example my customer would 10 of their products and let us collect data on these products. Since the manufacturing conditions varies from products to products, there's a natural grouping in my data.
Now my question is, when I do cross validation, should I do the train-validation split by groups or not? From sklearn documentation (http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-iterators-for-grouped-data), at first glance it seems to suggest I should since model made from a particular set of products should generalize to other products:

The i.i.d. assumption is broken if the underlying generative process yield groups of dependent samples.
Such a grouping of data is domain specific. An example would be when there is medical data collected from multiple patients, with multiple samples taken from each patient. And such data is likely to be dependent on the individual group. In our example, the patient id for each sample will be its group identifier.
In this case we would like to know if a model trained on a particular set of groups generalizes well to the unseen groups. To measure this, we need to ensure that all the samples in the validation fold come from groups that are not represented at all in the paired training fold.

However, when I think about it, splitting by group seems unnecessary. There is no hidden variable per se that would affect the spectra collected from different groups. In principle, the spectra are only determined by the structure on a product-this seems to be different from the example in the documentation, where supposedly the hidden variable is the patient itself (different individual can be more prone to some disease, or benefits more to some treatments). In fact, blind test on additional data seems to suggest that splitting by groups overestimate the cross validated MSE against splitting without groups.
 A: The domain specific qualification is everything here. On the one hand, you suggest that manufacturing conditions varies from products to products, but on the other hand the spectra are only determined by the structure on a product. Only you (or another domain expert) will know which one wins out here. 
I've used group kfold when I don't want observations from the same group to be in train and test sets, because the algorithm might overfit. Not sure if this applies for you.
A: 
the spectra are only determined by the structure on a product

Can you guarantee that production settings do have no whatsoever influence on structure? If so, you can neglect the groups. If production settings can possibly influence structure, you need to split by groups.

In fact, blind test on additional data seems to suggest that splitting by groups overestimate the cross validated MSE against splitting without groups.

In other words, if you split by groups, you see higher MSE? 
=> this is clear indication that you should split by groups. In other words, not splitting by groups will get you an overoptimistic error estimate. 
