I am a beginner in machine learning for time series, I need to develop a project, where my data is composed of minutes, could someone help me create this algorithm?

Data set: Each value represents one minute of collection (9:00, 9:01 ...), the collection lasts 10 minutes and was performed in 2 months, that is, 10 values ​​for January and 10 values for the month of February.

enter image description here

Complete data

Objective: I would like my result to be a forecast of the next 10 minutes for month of March, example:

2020-03-01 9:00:00
2020-03-01 9:01:00
2020-03-01 9:02:00
2020-03-01 9:03:00

Training: The training must contain the month of January and February as a reference for forecasting, taking into account that it is a time series


enter image description here


Current problem: it seems that the current forecast is failing, the previous data does not seem to be valid as a time series, because, as can be seen in the seasonality image, the data set is shown as a straight line. The forecast is represented by the green line in the figure below, and the original data by the blue line, however as we see the date axis is going until 2020-11-01, it should go until 2020-03-01, in addition the original data form a rectangle in the graph

enter image description here


# -*- coding: utf-8 -*-

    import pandas as pd
    import numpy as np
    import pmdarima as pm
    #%matplotlib inline
    import matplotlib.pyplot as plt
    from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
    from statsmodels.tsa.arima_model import ARIMA
    from statsmodels.tsa.seasonal import seasonal_decompose
    from dateutil.parser import parse
except ImportError as e:
    print("[FAILED] {}".format(e))

class operationsArima():

    def ForecastingWithArima():


            # Import
            data = pd.read_csv('minute.csv', parse_dates=['date'], index_col='date')

            # Plot
            fig, axes = plt.subplots(2, 1, figsize=(10,5), dpi=100, sharex=True)

            # Usual Differencing
            axes[0].plot(data[:], label='Original Series')
            axes[0].plot(data[:].diff(1), label='Usual Differencing')
            axes[0].set_title('Usual Differencing')
            axes[0].legend(loc='upper left', fontsize=10)
            print("[OK] Generated axes")

            # Seasonal
            axes[1].plot(data[:], label='Original Series')
            axes[1].plot(data[:].diff(11), label='Seasonal Differencing', color='green')
            axes[1].set_title('Seasonal Differencing')
            plt.legend(loc='upper left', fontsize=10)
            plt.suptitle('Drug Sales', fontsize=16)

            # Seasonal - fit stepwise auto-ARIMA
            smodel = pm.auto_arima(data, start_p=1, start_q=1,
                                    max_p=3, max_q=3, m=11,
                                    start_P=0, seasonal=True,
                                    d=None, D=1, trace=True,

            print("[OK] Generated model")

            # Forecast
            n_periods = 11
            fitted, confint = smodel.predict(n_periods=n_periods, return_conf_int=True)
            index_of_fc = pd.date_range(data.index[-1], periods = n_periods, freq='MS')

            # make series for plotting purpose
            fitted_series = pd.Series(fitted, index=index_of_fc)
            lower_series = pd.Series(confint[:, 0], index=index_of_fc)
            upper_series = pd.Series(confint[:, 1], index=index_of_fc)
            print("[OK] Generated series")

            # Plot
            plt.plot(fitted_series, color='darkgreen')
                            color='k', alpha=.15)

            plt.title("ARIMA - Final Forecast - Drug Sales")
            print("[SUCESS] Generated forecast")

        except Exception as e:

            print("[FAILED] Caused by: {}".format(e))

if __name__ == "__main__":
    flow = operationsArima()
    flow.ForecastingWithArima() # Init script


                                SARIMAX Results                                 
Dep. Variable:                        y   No. Observations:                   22
Model:             SARIMAX(0, 1, 0, 11)   Log Likelihood                     nan
Date:                  Mon, 13 Apr 2020   AIC                                nan
Time:                          21:19:10   BIC                                nan
Sample:                               0   HQIC                               nan
                                   - 22                                         
Covariance Type:                    opg                                         
                 coef    std err          z      P>|z|      [0.025      0.975]
intercept           0   5.33e-13          0      1.000   -1.05e-12    1.05e-12
sigma2          1e-10   5.81e-10      0.172      0.863   -1.04e-09    1.24e-09
Ljung-Box (Q):                         nan   Jarque-Bera (JB):                  nan
Prob(Q):                               nan   Prob(JB):                          nan
Heteroskedasticity (H):                nan   Skew:                              nan
Prob(H) (two-sided):                   nan   Kurtosis:                          nan

Your 22 values suggest two diametrically opposed viewpoints/approaches. You have 11 values per cyscle.

The first approach is to detect latent deterministic structure (i.e. unspecified causals ) ... in this case 7 of the 11 time points are statistically significant .

7 seasonal dummies reflecting 7 of the 11 periods were statistically significant enter image description here . The augmented data set is here enter image description here . The Actual/Fit and Forecast graph is here enter image description here . This approach identifies and augments the observed data with 10 dummies and finds three of them to be not-significant.

An alternate approach id to use the pure rear-window approach (arima) which simply believes that the past should be the basis for the forecast ignoring the fact that the past never causes the future and is only a proxy for omitted causal variables.

Using this approach the "found model" is here enter image description here where the value 11 periods is the best estimate of a future value.

Since the observed series is DETERMINISTIC , the resulting forecasts (BUT NOT INTERPRETATION) are identical.

My rational mind leaning on "causes" rather than simple memory strongly suggests that in this case approach 1 is not only the better strategy but it is sufficient , if not parsimonious.

My seasoned/general approach to time series modelling encompasses studying both approaches and seamlessly integrating both kinds of components when necessary along with any user-specified causal series.

I hope that this brings some clarity to your question although I couldn't exactly answer or duplicate you results.

  • $\begingroup$ your approach is as didactic as that of a teacher, excellent, congratulations, for sure every point of view of someone experienced can help, could you make available the code you used, tools and the like to perform such tests? $\endgroup$ – Luis Henrique Apr 14 '20 at 19:14
  • 1
    $\begingroup$ The "code" is AUTOBOX , available in R , however for the first approach simply create 10 seasonal dummies ..perform a regression ,, step-down and then you get my results. The second approach requires sophisticated arima model identification strategies ala autobox.com/pdfs/ARIMA%20FLOW%20CHART.pdf . Please accept my answer if you are as pleased as you say you are. $\endgroup$ – IrishStat Apr 14 '20 at 19:37
  • $\begingroup$ ok, very good, thanks a lot! $\endgroup$ – Luis Henrique Apr 15 '20 at 1:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.