The basis of testing the accuracy of any machine learning algorithm is to test the trained algorithm on data that it has never seen before. The usual approach to sample the test set is to just randomly select a subset from all observations.

However, I'm dealing with global spatial data (a regular sized raster) where a high degree of spatial autocorrelation exists, both in the features and the dependent variable. Therefore neighboring pixels tend to have almost identical feature and dependent variable values. If we just random sample the test set from all pixels the algorithm will already have seen the neighbors of the test pixels while training. As such the calculated test statistics will end up appearing better than they actually are (because, for all intents and purposes, the algorithm trained on the test data).

So is there a better way to sample the test set?

My current plan is to cut the whole data into large regular sized cells (each containing thousands of observations) and assign a part of these cells randomly to the test set. Then the test set has a higher spatial distance to the training data in average, reducing the effect of the autocorrelation. But on the other hand, this greatly increases the dependency within the training and validation set: If one prediction in the test set is wrong, chances are that the neighboring observations are predicted wrong too. But performance statistics like variance and bias of the residuals (for regression) or average/producers/users accuracy (for classification) should not be affected by this as long as the training data has a similar distribution than the testing data.

  • $\begingroup$ Note: The same problem also applies to selecting the validation set, I've omitted it for readabilitys sake. $\endgroup$ – Leander Moesinger Mar 27 '18 at 10:10
  • $\begingroup$ What will new data look like? I assume you don't have a panel? Or is there a time element to your data? $\endgroup$ – generic_user Mar 27 '18 at 11:09

For testing, you should try to recreate a scenario that is reasonably similar to your real-world use.

If you will only be be classifying randomly selected pixels, shuffling your training and testing data makes perfect sense. If however you will need to be able to correctly classify "large regular sized cells", then that's your testing scenario as your accuracy there will directly translate to your accuracy in a real-world scenario.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.