0
$\begingroup$

I'm studying validation and I've seen multiple examples of hold out validation. Some will hold the tail of the data, while others will leave out $n$ random points.

I assume it has to do with whether you're testing for interpolation or extrapolation, but is there something I'm missing as far as general practice goes? Because when you just leave out random points it just seems like 1 step of k-fold validation, without training and validating over the rest of the folds.

Thanks!

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ Selecting the tail of data is a terrible idea if your data is sorted in some systematic way that affects prediction. For example, if your data is sorted by date and you are modeling something that might be affected by date, experience, etc., holding out the most recent records might paint a worse (or better) picture of your model than selecting random records. $\endgroup$ – StatsStudent Jan 11 at 18:57
  • 1
    $\begingroup$ @StatsStudent Holding out the "tail" if they mean the most recent records (I would not call this tail though..) might make sense if you are comparing models intended to test on future data. In that case, testing against the most recent data is more like the real-world situation and might help you know if it makes sense to use a model that favors more recent data in fitting the model parameters. I agree in most cases it is a bad idea, though, just not necessarily terrible in every case. $\endgroup$ – Bryan Krause Jan 11 at 19:30
  • 1
    $\begingroup$ Agreed 100% Bryan. I think Yilun Zhang's answer below addresses this aspect well. $\endgroup$ – StatsStudent Jan 11 at 19:36
5
$\begingroup$

The point of hold out validation set is that you want part of your data to be left out from training so that you can test out the performance of your model on unseen data. Therefore, you need your validation set to have the same distribution as your training data. Thus, random selection is always the way to go since the data you use are obtained from somewhere else and you don't know how the data points are sorted, and you don't want the original sorting to affect which data goes into training and which goes into validation.

However, sometimes you do want to use the tail of the data. One obvious case is time series predictions. The data you have should usually be sorted by time. Thus, it makes a lot of sense to use the first X% as training data and use the remaining (100-X)% as validation since those are the future time points. If you did the random validation set split, you will be in trouble. For example, if values at time point 1,3, 4, 5 are in training and 2 is in validation, the model will be able to use time point 3, 4, 5 to infer the value at time point 2, which is completely not acceptable as you are leaking information.

How you select your validation set may vary depends on the objective and the modeling goal. However the principle is to make sure:

  1. The training and validation set are from the same distribution/population.
  2. Your training data does not leak information to your validation data.
|cite|improve this answer
$\endgroup$
  • $\begingroup$ Your explanation is very concise, but I'm not sure I understand the "leaking information" part. For example you have a time series from 2008 to 2018 and you want an estimate for a missing value, i.e. 2010. In that case it wouldn't make much sense leaving out only 2016,2017,2018, but random values in the range. Obviously if you want to predict 2019, it would make sense to leave the last values out. $\endgroup$ – George Jan 12 at 9:43
  • $\begingroup$ @George I think what I meant is that it's not reasonable to make a prediction for a time point in the middle of two training data points for time series data since the value can possibly be inferred from a future time point. Moreover, if you already have data point for $t_{2018}$, then why do you want to make a prediction for $t_{2017}$ ? $\endgroup$ – Yilun Zhang Jan 12 at 22:20

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.