Machine learning for time-series forecasting

Question

I am new to machine learning and have been trying to learn. I am forecasting AQI using ExtraTreesRegression model in Python. My training and testing results are coming good. While testing my model my r2 score is 90 but when I used the same model for the real-time application the r2 score degrade to 53. what is the issue taking place?

Another question is I am using train_test_split function to split my time-series data into train and test set. This function provides me shuffled data on which my testing r2 score is 90 but when I disable the shuffling option in the train_test_split function my r2 score degrade to 53. why this is taking place? How does the shuffling affects on time-series forecasting? Is shuffling is beneficial for time-series or not?

Answer 1

Yes, you can use regression algorithms for forecasting. There's a good explanation of how to adapt regression algorithms to forecasting problems here.

As stated in the comments, you need to make sure you properly evaluate your forecasting algorithms. When you use train_test_split you random shuffle and split your data. Instead you should only use past data to fit your algorithm and then evaluate against future data.

If you're interested, we're developing a toolbox that extends scikit-learn for exactly these use cases. So with sktime, you could simply write:

import numpy as np
from sktime.datasets import load_airline
from sktime.forecasting.compose import make_reduction
from sklearn.ensemble import ExtraTreesRegressor
from sktime.forecasting.model_selection import temporal_train_test_split
from sktime.performance_metrics.forecasting import mean_absolute_percentage_error

y = load_airline()  # load 1-dimensional time series
y_train, y_test = temporal_train_test_split(y)  
fh = np.arange(1, len(y_test) + 1)  # forecasting horizon
regressor = ExtraTreesRegressor()  
forecaster = make_reduction(regressor, window_length=10)
forecaster.fit(y_train)
y_pred = forecaster.predict(fh)
print(mean_absolute_percentage_error(y_test, y_pred))
>>> 0.17668623103710565