Reducing data set to preserve i.i.d. assumption in machine learning?

Question

In statistics there is a lot of literature about sampling, but I have not found any machine learning literature that discusses the quality/usefulness of the data set itself. This comes as a surprise to me, as the i.i.d. assumption is important in e.g. Vapnik–Chervonenkis theory for generalization.

Specifically, I have 88.000 data points for housing prices from the period 2000-2022. Assume that housing prices in 2000 resembles a different housing market than today's housing market, i.e. housing prices in 2000 are drawn from a different distribution. If I want to be able to predict housing prices today, is it an acceptable procedure to reduce the data set to avoid this problem? As there is a trade-off between data length and the i.i.d. assumption, I also thought of running two ML experiments, one in which the whole period is used, 2000-2020, and one in which e.g. 2016-2020 is used, and then use newly sold houses as test data. Once again, however, I'm unsure if this is the way to go.

Answer 1

It is not that much about independent and identically distributed random variables, because you can assume that the prices are not independent, but rather auto-correlated over time. In many cases, you would be interested in meeting the looser criterion of exchangability. Since it changes over time, you would rather be concerned if the underlying process is stationary.

But to answer this question you don't really need complicated statistical terms. In general, you want to have relevant data, that well represents the underlying distribution. Using irrelevant data is unlikely to help, it can even hurt the performance of the model. In some cases, you may be able to transfer what you've learned in one dataset to another (e.g. the seasonality in prices changes) so it still might be useful to include such data. But the general answer is that you should have good reasons to use the data if you know that it follows a different distribution than the prediction-time data. There is nothing bad in throwing away data that is irrelevant to your problem. As discussed by Xiao-Li Meng in the Statistical paradises and paradoxes in Big Data talk, you usually don't need more data, but better data.