7
$\begingroup$

I'm dealing with a forecasting problem that has a time component, similar to the following: I have 2 years of data on daily performance for thousands of agents, and want to predict the future performance per agent using an ensemble of decision trees (e.g. RandomForestRegressor, GBR, or XGBoost). My question deals with the optimal performance evaluation methodology employed when the underlying ground truth function is changing significantly over time (Cross-validation and OOB error are both inappropriate due to use of lag features).

I'm using TimeSeriesSplit as my methodology for both model performance estimation and hyperparameter selection. Optimal hyperparameters for earlier splits differ from later splits. Therefore, I want to emphasize performance of more recent predictions when tuning hyperparameters. (To exaggerate the problem, consider the use of stock market data over 20 years.) To do this, I believe it makes sense to weight the scores of recent splits higher; e.g. in a 15-fold split, performance of the first split (using the oldest data and smallest sample) would have the least weight, and the last split (which uses the most recent data and has the largest sample) would have the greatest weight.

timeseriessplit

A classic, naive approach to finding optimal hyperparameters using GridSearchCV or RandomizedSearchCV plus TimeSeriesSplit (sklearn) optimizes the model's hyperparameters for average performance weighted equally over all splits - from the first split to the last - but this is clearly not as good as sacrificing some accuracy in predictions from long ago if, as a tradeoff, we can sufficiently improve predictions on more-recent periods. Furthermore, the naive approach to hyperparameter tuning with TimeSeriesSplit will use the oldest data for training 15 times (for 15 splits), while the most-recent training data in evaluating model performance is only used once.

How should one choose split weights to implement such a "cross-validation" methodology (custom time-series split) in scikit-learn? I'd like to use this "weighted TimeSeriesSplit" in my pipeline.

$\endgroup$

closed as off-topic by usεr11852, Michael Chernick, Peter Flom Feb 3 at 13:21

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – usεr11852, Michael Chernick, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.