9
$\begingroup$

Consider a prediction setting in which we are interested in training a regression or classification function $f$ with inputs $X \in \mathbb{R}^k$ and target $Y$, and assessing its expected generalization performance on new data. For example, in a regression setting, we might want to estimate what our model's mean absolute error (MAE) or root mean square error (RMSE) will be on unseen data, and we typically do that by evaluating our model's predictions on a held-out test set.

Many standard textbooks (for example, Chapter 7, Model Assessment and Selection, of The Elements of Statistical Learning) discuss data partitioning and cross-validation in a context where we are dealing with i.i.d. samples from a joint distribution $F\left(X, Y\right)$. In this case, constructing the test set is straightforward: we take a simple random sample and call it our test set.

Are there any standard references that discuss how model assessment strategies should be modified when data are not i.i.d.? I've seen discussions that are specific to particular settings (for example, Data partitioning for spatial data is about spatial data), but I was wondering whether there is a general reference covering multiple settings.

Examples include:

  • Spatial data: the model may generalize more easily to points that are geographically close to the training set, and if we are interested in estimating the model's ability to generalize to points that are far from the training set, we'd need to account for that in our data partitioning / test set construction
  • Data with a natural discrete group or hierarchical structure: for example, if we are dealing with patients and hospitals, the question "how does my model generalize to new patients in a hospital that was included in the training set" differs from "how does my model generalize to new hospitals", and we should construct our test set(s) accordingly; we may even want to answer both questions, which we could do with two different test sets (one on held-out patients and another on held-out hospitals)
  • Similarly, if dealing with panel data (such as observing individuals over time), the question "How does my model do when predicting on an individual who was already observed $K$ times?" might have a different answer than "How does my model do on its first prediction for an individual who was never observed before?"
  • Time series data: in some time series contexts (see https://stats.stackexchange.com/a/195438/9330 for an exception), we want to construct our test sets such that they cover a period that is entirely "in the future" relative to the training set
  • More complex real-world examples might involve multiple "complications" relative to the simple i.i.d. case: we might be dealing with spatiotemporal data, for example, and it might also have a hierarchical or group structure

Does any textbook or paper discuss cross validation or test set construction in real-world problems, in a way that is general enough to cover all of the examples above (and possibly more)?

Related:

$\endgroup$
6
  • $\begingroup$ scikit-learn.org/stable/modules/… covers som of these topics (grouped data, time series -- but not spatial data). Does anyone have a good textbook reference that covers all of this in more detail? Or a paper? $\endgroup$
    – Adrian
    Feb 11, 2022 at 23:06
  • 1
    $\begingroup$ Excellent question that many people in practice pay too little attention to and I'd love more good references on this. I personally keep sharing this post fast.ai/2017/11/13/validation-sets and keep looking at what people do in places like Kaggle. The "How to win a data science competition" course on Coursera has some nice materials on this, too. $\endgroup$
    – Björn
    Feb 12, 2022 at 9:06
  • $\begingroup$ That post is definitely related to this question, thank you for the link. The example that says "you would be most interested in how the model performs on drivers you haven’t seen before" could fit into the panel data or group structure examples I gave above. $\endgroup$
    – Adrian
    Feb 14, 2022 at 22:33
  • 1
    $\begingroup$ This heavily depends on the goals and what you expect at deployment time. So it is domain specific and not really a machine learning topic. You need to split on anything that you want to claim generalization capacity for. So you must split on person, on sensor brand, source country, etc. depending on the use case. If you'll have the model trained on location-specific data on deployment, you don't need to split on location, as you don't care about generalization wrt location. Depending on the dataset, splitting on many things may force you to discard a lot of data. $\endgroup$
    – isarandi
    Feb 16, 2022 at 21:01
  • 1
    $\begingroup$ I agree that the answer will often be domain-specific, and that "you need to split on anything that you want to claim generalization capacity for" (that's a great way of putting it). But I don't see how that makes this "not a machine learning topic"! $\endgroup$
    – Adrian
    Feb 18, 2022 at 19:37

1 Answer 1

4
+50
$\begingroup$

It really all boils down to two rules of thumb:

  1. When splitting your data, leave out what you want to predict. If you want to generalize to new hospitals, rather than new patients at the same hospital, leave out one hospital at a time when doing CV — do not leave out one patient at a time, as this only tests your ability to generalize to patients at the same hospital.
  2. When doing cross-validation, split your test data into folds that can be considered approximately independent. For example, with time series data, you want to leave out a single run/“chunk” of observations at a time. If you have a time series running from 1900 to 2000 and want to use 10 folds, the first fold should be the first 10 years, the second the next 10, and so on. The idea here is that even if a time series isn’t independent, we can think of 10 years as enough for most of the correlation to disappear, especially if we’re comparing models that are already OK at dealing with the time series structure of our data. If we assign each year to a random fold, a model can easily “Cheat” by assuming that 2020 will look just the same as 2021 and 2019, but it’s hard to predict 2020 from 2010. A correlogram can help you identify how long a lag is enough that you can consider each block “Basically independent” of the others.

Some relevant papers:

https://onlinelibrary.wiley.com/doi/abs/10.1111/ecog.02881

https://www.sciencedirect.com/science/article/pii/S0020025511006773

https://www.tandfonline.com/doi/full/10.1080/00949655.2020.1783262

You can also check out the Sperrorest R package for this.

$\endgroup$
3
  • 1
    $\begingroup$ "Data often show temporal, spatial, hierarchical ... or phylogenetic structure. Modern statistical approaches are increasingly accounting for such dependencies. However, when performing cross-validation, these structures are regularly ignored, resulting in serious underestimation of predictive error. One cause for the poor performance of uncorrected (random) cross-validation, noted often by modellers, are dependence structures in the data that persist as dependence structures in model residuals, violating the assumption of independence." Exactly what I had in mind, thank you! $\endgroup$
    – Adrian
    Feb 22, 2022 at 18:45
  • 1
    $\begingroup$ Also related (although it's specific to spatial data): nature.com/articles/s41467-020-18321-y Spatial validation reveals poor predictive performance of large-scale ecological mapping models $\endgroup$
    – Adrian
    Feb 22, 2022 at 18:50
  • $\begingroup$ @Adrian forgot to mention -- you can also use time-series models in general and then look at the usual diagnostics like AIC or R^2. If you model your time series well, these measures will let you predict out-of-sample performance, because your data will be conditionally independent. For instance, an AR(1) model implies your data are independent, after correcting for a linear function of the previous value. $\endgroup$ Feb 23, 2022 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.