3
$\begingroup$

I have time series data for two and half year. I am performing xgb.cv (from Jan 2014 to March 2016) and its giving very good result, but when I am using new data (April 2016 to June 2016) to predict, the accuracy is getting worse. I am using R xgboost package. Any suggestion on improving accuracy?

$\endgroup$
  • $\begingroup$ Are you saying that the model works well on the data that was used to train it, but not as well on new data ? If so, that is to be expected. Further, one technique is to divide the training data into a training set and a validation set. Train on the training set and use the validation set to fine tune the model. Finally get accuracy on the test set. $\endgroup$ – aginensky Nov 19 '16 at 15:15
  • $\begingroup$ @aginensky: Will xgb.cv be good technique to tune the model or should we go from grid search or something else $\endgroup$ – user3713850 Nov 19 '16 at 15:17
  • $\begingroup$ I'm very far from an expert on the use of boosting with times series. However, in general, I'm a fan of using caret and it's parameters for tuning models. I'm not sure I understand your questions. For every possible value of the tuning parameter, typically one uses cv or something to estimate predictive accuracy. How does on pick the specific values that one will check with cv ? Grid search would be one obvious method. $\endgroup$ – aginensky Nov 20 '16 at 15:47
  • $\begingroup$ how many rows do you have? Can you point to a decent proxy/toy in publicly available data? In my limited experience, xgb loves lots of rows (tens of thousands) and a very low learning rate $(0.001 \gg x)$ for good generalization. $\endgroup$ – EngrStudent Mar 6 '17 at 17:36
3
$\begingroup$

It is likely that your features have time biases. Assume you want to predict month and year from your data. You would see very good performance during CV as you have enough data points at each month. However you can not predict the future(April 2016 to June 2016) very well using your predictor. Another example is that assume you have a feature that its density changes during the time. Then the difference between cv error and your prediction in the future would be big. To solve it, you can use following suggestions:

  • Drop time biased features: time related indices, features that has a big shift during time, etc
  • Create features that are partially independent of time: from date, get "day of week", "weekend" and so on
$\endgroup$
0
$\begingroup$

First, if there is a trend in time series, then tree-based model maybe not the good choice (because of tree model can't extrapolate, can't predict value bigger or smaller than the value in the training set), or you can remove the trend first, then using the xgboost to predict the residuals of linear models.

Second, as Mortezaaa suggests, if this time series correlation with day of week, holiday, weekends, or season and so on. Then you can construct many features to improve you prediction result! Beside it, the moving average of time series can be the features too. There is a lot of good example on kaggle, such as rossmann-store-sales prediction and bike-sharing-demand prediction, there are time series too, and the winners do a lot of feature engineering! You will learn a lot.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.