Literature on applying XGBoost to Time Series Data I'm currently working on doing a time-series model with very limited data. However, most of the independent variables I have are not time-dependent, cross-sectional data. As such I want to apply some form of regression or decision tree in such. Is there any literature, white paper, package or methodology where I can apply XGBoost or any of the regression trees or algorithms for such use?
 A: If I understand your question correctly, you have mixed time-series and cross-sectional data, or in other words, panel/longitudinal data with time-constant features. 
There are a number of ways how to handle this type of data in supervised learning. You can


*

*Reduce the time series data to cross-sectional data by 


*

*extracting features from the time series (using e.g. tsfresh) or 

*binning (e.g. treating each time point as a separate column,
essentially ignoring that they are ordered in time), once you have
purely cross-sectional data, you can directly apply regression
algorithms like XGBoost's regressor;


*Apply a dedicated time series algorithm to the time-series data and ensemble it with some algorithm for the cross-sectional data;

*Use bespoke algorithms that can handle mixed data internally (I'm not aware of any algorithm that's available off-the-shelf, but it's certainly possible in principle to do that). 


We're developing a toolbox for exactly these use cases that extends scikit-learn to time series data, it's called sktime. For example, for the binning approach, you could write: 
from sktime.datasets import load_gunpoint
from sktime.transformers.series_as_features.reduce import Tabularizer
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split
from xgboost import XGBClassifier

X, y = load_gunpoint(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y)
classifier = make_pipeline(Tabularizer(), XGBClassifier())
classifier.fit(X_train, y_train)
classifier.score(X_test, y_test)

A: It depends on exactly what you mean.
For example, if you mean something like you have multiple products and each product can belong to different categories so that category would be what you want to take into account: then 100% that's something where your best bet it try to featurize the time-series piece. Stuff like what @mloning is talking about or you can check out the M5 competition on Kaggle where the winners were all using boosted trees with features for week of year and stuff like that. You will lose some sense of the time-series piece but usually the extra features like product category are more important to control for so boosted trees will do well.
On the other hand, if you mean you have one product across time with features that line up like price then you could still try those examples but (especially since you mention limited data) you could also try out a package I am developing: ThymeBoost.
A very simple example of sales and marketing spend. The file is on the github but also can be found with a quick google.
import pandas as pd
from ThymeBoost import ThymeBoost as tb


df = pd.read_csv('Sales_and_Marketing.csv')
df.index = pd.to_datetime(df['Time Period'])
y = df['Sales']
X = df['Marketing Expense'].to_frame()

y_train, y_test = y.iloc[:-12], y.iloc[-12:]
X_train, X_test = X.iloc[:-12, :], X.iloc[-12:, :]

boosted_model = tb.ThymeBoost(verbose=1)
output = boosted_model.fit(y_train,
                           trend_estimator='linear',
                           fit_type='global',
                           seasonal_estimator='fourier',
                           exogenous_estimator='decision_tree',
                           global_cost='maicc',
                           seasonality_lr=.1,
                           exogenous_lr=.1,
                           tree_depth=2,
                           seasonal_period=12,
                           additive=False,
                           exogenous=X_train,
                           )



predicted_output = boosted_model.predict(output,
                                         forecast_horizon=12,
                                         future_exogenous=X_test)

predicted_output['predicted_exogenous'] = predicted_output['predicted_exogenous'] -1

plt.plot(y)
plt.plot(output['yhat'].append(predicted_output['predictions']))
plt.axvline(x=y.index[-13], color='red', linestyle='dashed')
plt.show()


So this is just using a simple linear trend + fourier basis function seasonality + decision tree (with a depth of 2, this is just scikit-learn's regression tree). ThymeBoost just combines gradient boosting and time-series decomposition so essentially the decision tree is itself a boosted tree which is interesting.
Of course this is in dev so it some settings are still unstable!
