# Problems with cross-validation for time-ordered data

Let's assume a simple scenario: You want to forecast the house prices of various properties. The dataset is cross-sectional in nature, as you only observe each property once. However, you also have a timestamp of the observation (month/year).

In a sense, this means that the data is intrinsically time-ordered. In general, I know that with time series data, the use of e.g., k-fold cross-validation can be problematic. However, I don't have real time series data (multiple obs. for one house), but time-ordered data across houses. It is also clear that for time series data, there are other approaches such as rolling estimation.

In particular, I would like to understand theoretically from a statistical/econometric perspective why this might be problematic rather than just intuitively. Does anyone have a theoretical guide to this topic or know of relevant literature to delve deeper into this topic?

EDIT: Based on @Björn's answer, it seems useful to provide more information about the goal here. The goal is actually to predict future prices based on the trained model.

• If there is ordering, major issue, naively applying standard cross-validation may create a situation of predicting past using future data points. We did a survey on cross-validation in time series and proposed approach to do full-cross validation. here. Though rolling estimation is standard practice. Commented Aug 2, 2021 at 10:07

Your validation splits should match your target task. If taking a database, in a randomized fashion finding out sales prices and then filling the numbers for the rest is the task (without an intent to predict future sales), then simple cross-validation is just fine. If the goal is to predict future prices, then a past-vs-future split is better (whether that's a single such split or multiple ones). Why? Here's some examples of what could go wrong (and it's usually very hard to intervene to prevent these):

• Multiple sales of the same house might be in the data. If I have a past and a future sale, then guessing the the current sales price was inbetween is a pretty good bet. Sufficiently complex models will manage to memorize data to give this kind of answer and you will not even notice until you specifically look for it (I've seen exactly this type of thing with a xgboost once). On the other hand, in practice you would not have this information and real predictions would be less reliable.
• Multiple similar new units might go onto the market at the same time. It's probably a good guess that identical properties in the same local go for about the same price (plus-minus a bit of negotiating skill by people).
• If things like inflation, state of the economy etc. change over time, then these might have various effects on property prices. If you have sales data at the exact same time of a future sale, then that might tell you a lot about these effects that you would not know in practice (even if you put current economic forecasts and inflation numbers into your model, which would presumably somewhat help).
• A model trained with future data does not need to extrapolate into the future, it only needs to interpolate. So, a validation set-up like that does not test the ability of a model to extrapolate into the future. Of course, if we plan to regularly re-train a model (let's say once a month), it will not ever need to extrapolate very far into the future, so this might be less of a concern. Nevertheless, extrapolating a little bit into the future is still harder than interpolating with knowledge of the future.
• All this, so far, assumed that individual sales don't affect each other, but that might also be the case.
• E.g. if people know the sales prices of recent sales in the area, they might target a price close to that (with adjustments depending on the property), but if that is so, then data from the future tells you something about similar past sales (but you would not have that knowledge when predicting the future).
• Or, sales prices in one area might move in the same way due to being affected by some local event (new highway being buildt, new school opened etc.) and the model would learn this via other future sales from the area rather than from the things that could predict this in practice (e.g. knowledge of local events that move house prices).

I can't speak to the theoretical aspects, but why not preprocess your data to avoid the problem? If you can get some house price inflation data you can inflate the prices of the older (earlier timestamped) properties to values you can reasonably expect them to fetch today and then train your predictive algorithm using apple to apple comparisons across the data.

• This seems natural in this setting with house prices, however, this is only an assumed simple scenario. In many cases, this will not be possible in reality.
– carl
Commented Jul 31, 2021 at 13:57

I don't think that having a time-stamp on the data is necessary problematic for housing price forecasting.

The procedure is to decompose the house into its attributes (size, location, etc.), add dummy variables for time and then do regression on these variables, this is called a "hedonic model" in the econometric literature, for a detailed discussion check for instance chapter 4.3 of "The practice of econometrics: Classic and contemporary" Berdnt, 1996.

When you use this method, you estimate the coefficients accompanying the dummy time variables, so the model does not use specific house values from the past to compute future values.

One critical issue is the cage of covariance includes time varying effects, which can impose and unrecognised pattern on the data if not handled appropriately. There will be a general trends across house prices depending on the general economic situation evolving over time. more complex patterns such as evolving fashions and functional requirements will impose addition layers of complexity.

It is related to the Moiré effect. Here's a figure I've created recently that shows how hidden patterns in the data can affect CV, using an extreme example for demonstration purposes. Moiré patterns can be extremely complex and hard to detect if you don't look for them intentionally. If the periodicity of the sampling is a multiplicative factor of the periodicity of the underlying trend then you may under-sample underlying effects in the modelling set that mostly get held out for the test set (or vice versa).

Visualisation of experimental factor distribution in the simulated structured dataset. The color is created by red channel for time interval, green channel to indicate dose exposure, blue channel to indicate experimental group membership. Selection to hold out group indicated 1/3 reduction in intensity. a-c) Week of study as x-axis, dose group (sub-divided by experimental group) as Y-axis. a) is the basic experimental design matrix. b) black pixels indicate samples selected by sequential block selection for K-fold c) black pixels indicate samples selected using Monte-Carlo d) the average properties of the train and test sets with each selection method.