0
$\begingroup$

I am new to machine learning and have been trying to learn. I am forecasting AQI using ExtraTreesRegression model in Python. My training and testing results are coming good. While testing my model my r2 score is 90 but when I used the same model for the real-time application the r2 score degrade to 53. what is the issue taking place?

Another question is I am using train_test_split function to split my time-series data into train and test set. This function provides me shuffled data on which my testing r2 score is 90 but when I disable the shuffling option in the train_test_split function my r2 score degrade to 53. why this is taking place? How does the shuffling affects on time-series forecasting? Is shuffling is beneficial for time-series or not?

$\endgroup$
4
  • 2
    $\begingroup$ Shuffling a dataset for time-series forecasting is not a good practice. It would be helpful if you could provide an MCV of your issue and data. $\endgroup$
    – Ic3fr0g
    May 19, 2020 at 5:06
  • 2
    $\begingroup$ Use a TimeseriesSplitter instead of train_test_split. Your performance now is overestimated, as you have data leakage $\endgroup$
    – Jon Nordby
    May 19, 2020 at 7:07
  • $\begingroup$ @jonnor thank you for your reply. $\endgroup$ May 20, 2020 at 13:57
  • $\begingroup$ @Ic3fr0g thank you for your reply. $\endgroup$ May 20, 2020 at 13:58

1 Answer 1

1
$\begingroup$

Yes, you can use regression algorithms for forecasting. There's a good explanation of how to adapt regression algorithms to forecasting problems here.

As stated in the comments, you need to make sure you properly evaluate your forecasting algorithms. When you use train_test_split you random shuffle and split your data. Instead you should only use past data to fit your algorithm and then evaluate against future data.

If you're interested, we're developing a toolbox that extends scikit-learn for exactly these use cases. So with sktime, you could simply write:

import numpy as np
from sktime.datasets import load_airline
from sktime.forecasting.compose import make_reduction
from sklearn.ensemble import ExtraTreesRegressor
from sktime.forecasting.model_selection import temporal_train_test_split
from sktime.performance_metrics.forecasting import mean_absolute_percentage_error

y = load_airline()  # load 1-dimensional time series
y_train, y_test = temporal_train_test_split(y)  
fh = np.arange(1, len(y_test) + 1)  # forecasting horizon
regressor = ExtraTreesRegressor()  
forecaster = make_reduction(regressor, window_length=10)
forecaster.fit(y_train)
y_pred = forecaster.predict(fh)
print(mean_absolute_percentage_error(y_test, y_pred))
>>> 0.17668623103710565
$\endgroup$
2
  • $\begingroup$ That first link does not work! $\endgroup$
    – Tylerr
    Jun 18, 2021 at 13:26
  • 1
    $\begingroup$ @Tylerr I fixed the link and updated the code too to reflect the most recent release. $\endgroup$
    – mloning
    Jun 18, 2021 at 17:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.