Preventing Data Leakage in Time Series Forecasting with Feature Engineering

Question

In a previous question (linked here), I sought guidance on forecasting thousands of time series. Based on the suggestion to treat it as a regression problem, I used the LightGBM model with extensive feature engineering.

Feature engineering involved:

1. Date features extraction (e.g., month, year, day_of_year) using Pandas Date time Component.
2. Lag features with various lag periods.

lag_list = [1, 2, 3, 4, 5, 7, 9, 10, 14, 30, 60, 90]
forecasting_goal = 'Hours'
for lag in lag_list:
   df['l' + str(lag)] = df.groupby(["TeamID", "TaskID"])[forecasting_goal].transform(lambda x: x.shift(lag))

3. Rolling features with different window types and sizes.

window_types = ['triang', 'parzen', 'nuttall', 'hann', 'cosine', 'barthann']
window_sizes= [5, 7, 14, 30]
lag_list= [1, 2, 3, 4, 5, 7]
forecasting_goal = 'Hours'
    for lag in lag_list:
        for size in window_sizes:
                df['l' + str(lag) + '_rme' + str(size)] = df.groupby(["TeamID", "TaskID"])[forecasting_goal]\
                    .transform(lambda x: x.shift(lag).rolling(window=size).mean())

                df['l' + str(lag) + '_std' + str(size)] = df.groupby(["TeamID", "TaskID"])[forecasting_goal]\
                    .transform(lambda x: x.shift(lag).rolling(window=size).std())
                
                df['l' + str(lag) + '_rmx' + str(size)] = df.groupby(["TeamID", "TaskID"])[forecasting_goal]\
                    .transform(lambda x: x.shift(lag).rolling(window=size).max())

                df['l' + str(lag) + '_rmn' + str(size)] = df.groupby(["TeamID", "TaskID"])[forecasting_goal]\
                    .transform(lambda x: x.shift(lag).rolling(window=size).min())
            
            for typ in window_types:
                df['l' + str(lag) + '_rme' + str(size) + '_' + typ] = df.groupby(["TeamID", "TaskID"])[forecasting_goal]\
                    .transform(lambda x: x.shift(lag).rolling(window=size, win_type=typ).mean())

4. Exponential weighted moving average features.

alphas= [0.3, 0.5, 0.7, 0.8, 0.9]
lag_list= [1, 2, 3, 4, 5, 7]
forecasting_goal= 'Hours'
for lag in lag_list:
        for alpha in alphas:
            df['l' + str(lag) + '_ewma_' + 'alp' + str(alpha).replace(".", "")] = df.groupby(["TeamID", "TaskID"])[forecasting_goal]\
                .transform(lambda x: x.shift(lag).ewm(alpha=alpha).mean())

5. Expanding mean features.

forecasting_goal = 'Hours'
expand_list= [1, 3, 7]
lag_list= [1, 2, 3, 4, 5, 7]

for exp_mean in expand_list:
    for lag in lag_list:
        df['l' + str(lag) + '_exp' + str(exp_mean)] = df.groupby(["TeamID", "TaskID"])[forecasting_goal]\
                .transform(lambda x: x.shift(lag).expanding(min_periods=exp_mean).mean())

6. Periodic features like Sin and Cos waves.

The model performed well until I discovered potential data leakage in the testing set. I realized that using actual hours from the testing set to calculate rolling features might cause leakage because, in real-world scenarios, future actual hours are unknown.

I'm seeking advice on how to address this issue, especially when forecasting multiple months ahead.

How can I prevent data leakage and ensure accurate time series forecasting in such a scenario?

Answer 1

Firstly, it is optional that this is not a leakage - depends on your production environment: If "in production" you need to predict only 1 step ahead- you're all good. You use all the history, including the previous day or so, and predict the next. which is fine.

On the other hand, if "in production" you would like to predict months ahead - you indeed have a leakage. The way you should handle it is by modeling the task as predicting the following X values. Some models can do that (depends on the data of course).

Secondly, lightGBM is a great model, but it has some disadvantages when dealing with time series forecasting, versus other models created for these kinds of tasks.

I'd start by considering an easier task than predicting numeric values months ahead. It's a tough problem, if possible at all.

For any farther questions, feel free to comment and I'd try my best to assist.