i am building a model that predicts likelihood of the presence of an infectious disease across 10000 villages. There is spatial dimension - each villages is surrounded by e.g. 8 other villages, and presence of disease in a nearby village increased the likelihood of disease in the village in question.

I created a feature that sums the number of cases in the 8 nearby villages. However - initially - i created this value on the entire dataset. With regards to e.g. 10 fold cross validation - is this "wrong"? Am i bringing in information from "outside the fold" and hence bias the performance? Should i run the feature creation within each fold, even those some of the nearby villages will be outside the fold? (also - computationally this might become very heavy...)

Thoughts welcome - much appreciated


Yes, this will potentially introduce a bias in the performance estimates, so I would recommend repeating the calculation independently in each fold. It is best to regard cross-validation as a means of estimating the performance of a method of generating a model, rather than of the model itself, so you need to perform every step of that method independently in each fold in order to get an (approximately) unbiased performance estimate.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.