The basis of testing the accuracy of any machine learning algorithm is to test the trained algorithm on data that it has never seen before. The usual approach to sample the test set is to just randomly select a subset from all observations.
However, I'm dealing with global spatial data (a regular sized raster) where a high degree of spatial autocorrelation exists, both in the features and the dependent variable. Therefore neighboring pixels tend to have almost identical feature and dependent variable values. If we just random sample the test set from all pixels the algorithm will already have seen the neighbors of the test pixels while training. As such the calculated test statistics will end up appearing better than they actually are (because, for all intents and purposes, the algorithm trained on the test data).
So is there a better way to sample the test set?
My current plan is to cut the whole data into large regular sized cells (each containing thousands of observations) and assign a part of these cells randomly to the test set. Then the test set has a higher spatial distance to the training data in average, reducing the effect of the autocorrelation. But on the other hand, this greatly increases the dependency within the training and validation set: If one prediction in the test set is wrong, chances are that the neighboring observations are predicted wrong too. But performance statistics like variance and bias of the residuals (for regression) or average/producers/users accuracy (for classification) should not be affected by this as long as the training data has a similar distribution than the testing data.
