Should out-of-sample validation also be out-of-time for time-series?

Question

Introduction
When training a model a "sample" usually refers to the data used to fit the model, so...

Sample: Data used for training model
Out-of-sample: Data not used for training model
Out-of-time: Data not used for training model that is later than data used to train the model

Sometimes regulation says you must perform "out-of-sample" and "out-of-time" validation of your model. But it is not clear what is ment as "out-of-time" is already "out-of-sample". I think "out-of-sample" could be interpreted as validation on customers that were not part of the training "sample".

Interpretation 1
Here the interpretation is that one makes two independent validation runs:

1. Validation on "out-of-sample" data that is "in-time"
2. Validation on "out-of-time" data that is also "out-of-sample"

The big flaw as I see it here is that the "out-of-sample" validation will be "in-time", thus the performance metrics will likely be very inflated. An example would be if the "in-time" period was during covid, then you would get "out-of-sample" validation results indicating that you are doing great on these unseen customers, while in reality the model would perform horribly on the unseen customers in a production environment (which is out-of-time).

Interpretation 2
Here the interpretation is that it makes little sense to do any reporting of numbers from the "in-time" "out-of-sample" data, as the numbers don't really say anything about real world performance. Instead all validation numbers reported to the regulators are out-of-time.

1. Validation on "out-of-sample" data that is "out-of-time" alone. This is the MOST conservative estimate of real world performance. It is also the best metric for generalizability. Here "out-of-sample" refers to customers, who have not even been in the training sample "in-time".
2. Validation on ALL "out-of-time" data, as this is the best estimate of performance in the immediate future, as a customer base is usually quite stable in the immediate future (e.g. the next 2-3 years).

With this interpretation that numbers you report are the most conservative, likely reflect real-world performance the best and reflect generalizability (e.g. to unseen customers).

Question Which interpretation would you use? Do you think both interpretations are equally valid?

Answer 1

For precisely the reasons you outline, validation and testing data should always respect the time ordering and not "learn from the future", because real world performance (I like to call this "in production") is ultimately what we care about. I would argue that this may not only apply to cases commonly called "forecasting", but also to "classification" tasks with time dimensions: if you build a model for credit scoring or churn prediction, the time dynamics may be so strong that you may be well advised to include the time dimension in your modeling and only use out-of-time test/validation sets.

Now, this is to a degree opinion-based.

However, this is in my opinion the absolute consensus in the forecasting community (less so among classifiers), which I have been involved with for a number of years, so yes, I do believe I know what I am talking about. As an Associate Editor and frequent reviewer at the International Journal of Forecasting, any submission that "learns from the future" will get a major revision at the very least, or a reject usually, because the authors apparently did not understand the very first thing about forecasting. Finally, all serious forecasting resources will be very clear on this point.