0
$\begingroup$

I am working in manufacturing related industry. I am tasked with building some models to predict some physical quantities (for example, size of a hole in the structure) from spectra. The data usually comes in groups, for example my customer would 10 of their products and let us collect data on these products. Since the manufacturing conditions varies from products to products, there's a natural grouping in my data.

Now my question is, when I do cross validation, should I do the train-validation split by groups or not? From sklearn documentation (http://scikit-learn.org/stable/modules/cross_validation.html#cross-validation-iterators-for-grouped-data), at first glance it seems to suggest I should since model made from a particular set of products should generalize to other products:

The i.i.d. assumption is broken if the underlying generative process yield groups of dependent samples.

Such a grouping of data is domain specific. An example would be when there is medical data collected from multiple patients, with multiple samples taken from each patient. And such data is likely to be dependent on the individual group. In our example, the patient id for each sample will be its group identifier.

In this case we would like to know if a model trained on a particular set of groups generalizes well to the unseen groups. To measure this, we need to ensure that all the samples in the validation fold come from groups that are not represented at all in the paired training fold.

However, when I think about it, splitting by group seems unnecessary. There is no hidden variable per se that would affect the spectra collected from different groups. In principle, the spectra are only determined by the structure on a product-this seems to be different from the example in the documentation, where supposedly the hidden variable is the patient itself (different individual can be more prone to some disease, or benefits more to some treatments). In fact, blind test on additional data seems to suggest that splitting by groups overestimate the cross validated MSE against splitting without groups.

$\endgroup$
1
$\begingroup$

the spectra are only determined by the structure on a product

Can you guarantee that production settings do have no whatsoever influence on structure? If so, you can neglect the groups. If production settings can possibly influence structure, you need to split by groups.

In fact, blind test on additional data seems to suggest that splitting by groups overestimate the cross validated MSE against splitting without groups.

In other words, if you split by groups, you see higher MSE? => this is clear indication that you should split by groups. In other words, not splitting by groups will get you an overoptimistic error estimate.

$\endgroup$
  • $\begingroup$ In my case, splitting by group give me an overpessimistic error estimate. I reserved a subset of data for blind test (i.e. not seen at all by the training process). I ran the CV process with Group-KFold and KFold to obtain 2 different models. The Group-KFold model underfit (MSE on blind test is larger than the KFold model), while both models have no overfitting issue (Training MSE is about the same as blind test MSE). $\endgroup$ – lizardfireman Jan 24 at 2:00
  • $\begingroup$ @lizardfireman: was that blind test set set aside splitting groupwise? $\endgroup$ – cbeleites supports Monica Jan 24 at 14:13
  • 1
    $\begingroup$ I have 30+ groups and I left about 10 as blind test, not sure if this is what you meant by splitting groupwise. The blind test data does not contain any data from the groups that are in the training set. $\endgroup$ – lizardfireman Jan 29 at 5:52
0
$\begingroup$

The domain specific qualification is everything here. On the one hand, you suggest that manufacturing conditions varies from products to products, but on the other hand the spectra are only determined by the structure on a product. Only you (or another domain expert) will know which one wins out here.

I've used group kfold when I don't want observations from the same group to be in train and test sets, because the algorithm might overfit. Not sure if this applies for you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.