0
$\begingroup$

I'm trying to create a forecsast with ARIMA(3, 1, 1) and the forecast is giving me a straight line for out of sample predictions.

enter image description here

Time Series Decomposition of original dataset: enter image description here

Original data set:

Date    Count
10/30/2018  4
10/31/2018  13
11/1/2018   11
11/2/2018   14
11/3/2018   7
11/4/2018   12
11/5/2018   7
11/6/2018   9
11/7/2018   4
11/8/2018   1
11/9/2018   4
11/10/2018  5
11/11/2018  9
11/12/2018  4
11/13/2018  6
11/14/2018  2
11/15/2018  3
11/16/2018  8
11/17/2018  4
11/18/2018  4
11/19/2018  7
11/20/2018  5
11/21/2018  7
11/22/2018  5
11/23/2018  7
11/24/2018  5
11/25/2018  5
11/26/2018  15
11/27/2018  4
11/28/2018  2
11/29/2018  7
11/30/2018  10
12/1/2018   2
12/2/2018   7
12/3/2018   3
12/4/2018   3
12/5/2018   2
12/6/2018   8
12/7/2018   6.5
12/8/2018   5
12/9/2018   5
12/10/2018  4
12/11/2018  1
12/12/2018  5
12/13/2018  4
12/14/2018  1
12/15/2018  1
12/16/2018  7
12/17/2018  2
12/18/2018  3
12/19/2018  1
12/20/2018  1
12/21/2018  2
12/22/2018  6
12/23/2018  4
12/24/2018  5
12/25/2018  2
12/26/2018  1
12/27/2018  3
12/28/2018  6
12/29/2018  4
12/30/2018  4
12/31/2018  12
1/1/2019    10
1/2/2019    2
1/3/2019    8
1/4/2019    3
1/5/2019    4
1/6/2019    2
1/7/2019    4
1/8/2019    7
1/9/2019    5
1/10/2019   3
1/11/2019   7
1/12/2019   41
1/13/2019   2
1/14/2019   17
1/15/2019   10.5
1/16/2019   4
1/17/2019   9
1/18/2019   5
1/19/2019   4
1/20/2019   9
1/21/2019   3
1/22/2019   3
1/23/2019   5
1/24/2019   2
1/25/2019   1
1/26/2019   1
1/27/2019   1
1/28/2019   3
1/29/2019   6
1/30/2019   3
1/31/2019   6
2/1/2019    38
2/2/2019    3
2/3/2019    17
2/4/2019    7
2/5/2019    3
2/6/2019    6
2/7/2019    4
2/8/2019    3
2/9/2019    5
2/10/2019   5
2/11/2019   5
2/12/2019   8
2/13/2019   3
2/14/2019   4
2/15/2019   8
2/16/2019   1
2/17/2019   1
2/18/2019   3
2/19/2019   4
2/20/2019   4
2/21/2019   5
2/22/2019   6
2/23/2019   1
2/24/2019   4
2/25/2019   1
2/26/2019   5
2/27/2019   3
2/28/2019   2
3/1/2019    4
3/2/2019    5
3/3/2019    1
3/4/2019    2
3/5/2019    6
3/6/2019    3
3/7/2019    1
3/8/2019    3
3/9/2019    3
3/10/2019   3
3/11/2019   6
3/12/2019   3
3/13/2019   3
3/14/2019   8
3/15/2019   9
3/16/2019   6
3/17/2019   4
3/18/2019   4
3/19/2019   2
3/20/2019   2
3/21/2019   4
3/22/2019   7
3/23/2019   9
3/24/2019   6
3/25/2019   2
3/26/2019   5
3/27/2019   6
3/28/2019   3
3/29/2019   6
3/30/2019   7
3/31/2019   4
4/1/2019    1
4/2/2019    4
4/3/2019    4
4/4/2019    6
4/5/2019    2
4/6/2019    2
4/7/2019    2
4/8/2019    4
4/9/2019    6
4/10/2019   7
4/11/2019   5
4/12/2019   5
4/13/2019   3
4/14/2019   1
4/15/2019   1.060606061
4/16/2019   1.121212121
4/17/2019   1.181818182
4/18/2019   1.242424242
4/19/2019   1.303030303
4/20/2019   1.363636364
4/21/2019   1.424242424
4/22/2019   1.484848485
4/23/2019   1.545454545
4/24/2019   1.606060606
4/25/2019   1.666666667
4/26/2019   1.727272727
4/27/2019   1.787878788
4/28/2019   1.848484848
4/29/2019   1.909090909
4/30/2019   1.96969697
5/1/2019    2.03030303
5/2/2019    2.090909091
5/3/2019    2.151515152
5/4/2019    2.212121212
5/5/2019    2.272727273
5/6/2019    2.333333333
5/7/2019    2.393939394
5/8/2019    2.454545455
5/9/2019    2.515151515
5/10/2019   2.575757576
5/11/2019   2.636363636
5/12/2019   2.696969697
5/13/2019   2.757575758
5/14/2019   2.818181818
5/15/2019   2.878787879
5/16/2019   2.939393939
5/17/2019   3
5/18/2019   3.060606061
5/19/2019   3.121212121
5/20/2019   3.181818182
5/21/2019   3.242424242
5/22/2019   3.303030303
5/23/2019   3.363636364
5/24/2019   3.424242424
5/25/2019   3.484848485
5/26/2019   3.545454545
5/27/2019   3.606060606
5/28/2019   3.666666667
5/29/2019   3.727272727
5/30/2019   3.787878788
5/31/2019   3.848484848
6/1/2019    3.909090909
6/2/2019    3.96969697
6/3/2019    4.03030303
6/4/2019    4.090909091
6/5/2019    4.151515152
6/6/2019    4.212121212
6/7/2019    4.272727273
6/8/2019    4.333333333
6/9/2019    4.393939394
6/10/2019   4.454545455
6/11/2019   4.515151515
6/12/2019   4.575757576
6/13/2019   4.636363636
6/14/2019   4.696969697
6/15/2019   4.757575758
6/16/2019   4.818181818
6/17/2019   4.878787879
6/18/2019   4.939393939
6/19/2019   5
6/20/2019   5.060606061
6/21/2019   5.121212121
6/22/2019   5.181818182
6/23/2019   5.242424242
6/24/2019   5.303030303
6/25/2019   5.363636364
6/26/2019   5.424242424
6/27/2019   5.484848485
6/28/2019   5.545454545
6/29/2019   5.606060606
6/30/2019   5.666666667
7/1/2019    5.727272727
7/2/2019    5.787878788
7/3/2019    5.848484848
7/4/2019    5.909090909
7/5/2019    5.96969697
7/6/2019    6.03030303
7/7/2019    6.090909091
7/8/2019    6.151515152
7/9/2019    6.212121212
7/10/2019   6.272727273
7/11/2019   6.333333333
7/12/2019   6.393939394
7/13/2019   6.454545455
7/14/2019   6.515151515
7/15/2019   6.575757576
7/16/2019   6.636363636
7/17/2019   6.696969697
7/18/2019   6.757575758
7/19/2019   6.818181818
7/20/2019   6.878787879
7/21/2019   6.939393939
7/22/2019   7
7/23/2019   7.060606061
7/24/2019   7.121212121
7/25/2019   7.181818182
7/26/2019   7.242424242
7/27/2019   7.303030303
7/28/2019   7.363636364
7/29/2019   7.424242424
7/30/2019   7.484848485
7/31/2019   7.545454545
8/1/2019    7.606060606
8/2/2019    7.666666667
8/3/2019    7.727272727
8/4/2019    7.787878788
8/5/2019    7.848484848
8/6/2019    7.909090909
8/7/2019    7.96969697
8/8/2019    8.03030303
8/9/2019    8.090909091
8/10/2019   8.151515152
8/11/2019   8.212121212
8/12/2019   8.272727273
8/13/2019   8.333333333
8/14/2019   8.393939394
8/15/2019   8.454545455
8/16/2019   8.515151515
8/17/2019   8.575757576
8/18/2019   8.636363636
8/19/2019   8.696969697
8/20/2019   8.757575758
8/21/2019   8.818181818
8/22/2019   8.878787879
8/23/2019   8.939393939
8/24/2019   9
8/25/2019   9.060606061
8/26/2019   9.121212121
8/27/2019   9.181818182
8/28/2019   9.242424242
8/29/2019   9.303030303
8/30/2019   9.363636364
8/31/2019   9.424242424
9/1/2019    9.484848485
9/2/2019    9.545454545
9/3/2019    9.606060606
9/4/2019    9.666666667
9/5/2019    9.727272727
9/6/2019    9.787878788
9/7/2019    9.848484848
9/8/2019    9.909090909
9/9/2019    9.96969697
9/10/2019   10.03030303
9/11/2019   10.09090909
9/12/2019   10.15151515
9/13/2019   10.21212121
9/14/2019   10.27272727
9/15/2019   10.33333333
9/16/2019   10.39393939
9/17/2019   10.45454545
9/18/2019   10.51515152
9/19/2019   10.57575758
9/20/2019   10.63636364
9/21/2019   10.6969697
9/22/2019   10.75757576
9/23/2019   10.81818182
9/24/2019   10.87878788
9/25/2019   10.93939394
9/26/2019   11
9/27/2019   11.06060606
9/28/2019   11.12121212
9/29/2019   11.18181818
9/30/2019   11.24242424
10/1/2019   11.3030303
10/2/2019   11.36363636
10/3/2019   11.42424242
10/4/2019   11.48484848
10/5/2019   11.54545455
10/6/2019   11.60606061
10/7/2019   11.66666667
10/8/2019   11.72727273
10/9/2019   11.78787879
10/10/2019  11.84848485
10/11/2019  11.90909091
10/12/2019  11.96969697
10/13/2019  12.03030303
10/14/2019  12.09090909
10/15/2019  12.15151515
10/16/2019  12.21212121
10/17/2019  12.27272727
10/18/2019  12.33333333
10/19/2019  12.39393939
10/20/2019  12.45454545
10/21/2019  12.51515152
10/22/2019  12.57575758
10/23/2019  12.63636364
10/24/2019  12.6969697
10/25/2019  12.75757576
10/26/2019  12.81818182
10/27/2019  12.87878788
10/28/2019  12.93939394
10/29/2019  13
10/30/2019  7
10/31/2019  2
11/1/2019   6
11/2/2019   1
11/3/2019   8
11/4/2019   5
11/5/2019   14
11/6/2019   4
11/7/2019   11
11/8/2019   15
11/9/2019   6
11/10/2019  9
11/11/2019  10
11/12/2019  6
11/13/2019  2
11/14/2019  4
11/15/2019  6
11/16/2019  1
11/17/2019  10
11/18/2019  3
11/19/2019  4
11/20/2019  5
11/21/2019  3
11/22/2019  4
11/23/2019  3.5
11/24/2019  3
11/25/2019  4
11/26/2019  10
11/27/2019  5
11/28/2019  8
11/29/2019  4
11/30/2019  1
12/1/2019   1
12/2/2019   4
12/3/2019   4
12/4/2019   9
12/5/2019   3
12/6/2019   3
12/7/2019   7
12/8/2019   7
12/9/2019   5
12/10/2019  7
12/11/2019  8
12/12/2019  5
12/13/2019  7
12/14/2019  11
12/15/2019  5
12/16/2019  8
12/17/2019  7
12/18/2019  9
12/19/2019  4
12/20/2019  4
12/21/2019  11
12/22/2019  7
12/23/2019  7
12/24/2019  5
12/25/2019  6
12/26/2019  4
12/27/2019  5
12/28/2019  10
12/29/2019  12
12/30/2019  4
12/31/2019  5
1/1/2020    9
1/2/2020    6
1/3/2020    8
1/4/2020    8
1/5/2020    6
1/6/2020    6
1/7/2020    5
1/8/2020    9
1/9/2020    5
1/10/2020   14
1/11/2020   4
1/12/2020   4
1/13/2020   8
1/14/2020   8
1/15/2020   14
1/16/2020   10
1/17/2020   4
1/18/2020   1
1/19/2020   4
1/20/2020   5
1/21/2020   3
1/22/2020   4
1/23/2020   2
1/24/2020   9
1/25/2020   2
1/26/2020   9
1/27/2020   2
1/28/2020   7
1/29/2020   2
1/30/2020   5
1/31/2020   7
2/1/2020    6
2/2/2020    1
2/3/2020    5
2/4/2020    6
2/5/2020    9
2/6/2020    5
2/7/2020    3
2/8/2020    4
2/9/2020    2
2/10/2020   6
2/11/2020   4
2/12/2020   6
2/13/2020   1
2/14/2020   3
2/15/2020   6
2/16/2020   1
2/17/2020   2
2/18/2020   6
2/19/2020   2
2/20/2020   3
2/21/2020   4
2/22/2020   3
2/23/2020   6
2/24/2020   5
2/25/2020   7
2/26/2020   9
2/27/2020   2
2/28/2020   5
2/29/2020   5
3/1/2020    5
3/2/2020    6
3/3/2020    1
3/4/2020    4
3/5/2020    4
3/6/2020    2
3/7/2020    6
3/8/2020    1
3/9/2020    1
3/10/2020   2
3/11/2020   3
3/12/2020   3
3/13/2020   1
3/14/2020   1
3/15/2020   1

The straight line between May 2019 and October 2019 is because we don't have data for that period but I don't need the forecast for that period. Just to make sure that this doesn't create any issues I have interpolated the missing values:

df_Carnaval=df_Carnaval.asfreq('d') 
df_Carnaval['Count'].interpolate(method='time', inplace=True)

To make the time series stationary, I have applied box cox transformation and taken 1st order differencing(d=1):

from sklearn.preprocessing import PowerTransformer
pt=PowerTransformer(method='box-cox')
Carnaval_transformed=df_Carnaval.copy()
Carnaval_transformed['Count']=pt.fit_transform(df_Carnaval.Count.values.reshape(-1,1))
Carnaval_transformed_diff = Carnaval_transformed.Count - Carnaval_transformed.Count.shift()

The PACF and ACF plot for the transformed and first order differenced data: enter image description here

I tried doing the forecasting with Prophet after applying box cox transformation but as you can see from the graph below, Prophet did not fit well. enter image description here

$\endgroup$
4
  • $\begingroup$ Interpolation was probably not a good idea, it must have messed up the estimated coefficients considerably. But the flat forecast is not because of that – this is simply the best long-term forecast for a process like ARIMA(3,1,1). $\endgroup$ – Richard Hardy Aug 17 '20 at 7:25
  • $\begingroup$ If I don't interpolate, there are some null values created after I set the frequency as 'daily'. i.e. right after this line "df_Carnaval=df_Carnaval.asfreq('d')" I get 250 missing values. If this is the best long term forecast for ARIMA (3,1,1), is there anything else I can try? Because Prophet seems to give me better results then and I'll go with that. $\endgroup$ – Aastha Jha Aug 17 '20 at 7:38
  • $\begingroup$ If your process is indeed well approximated by an ARIMA(3,1,1), following Prophet instead of ARIMA will actually yield worse results. The important question is, what model approximates the reality best. $\endgroup$ – Richard Hardy Aug 17 '20 at 8:56
  • $\begingroup$ To me, there doesn't seem to be any seasonality in the dataset in the ACF plot. Plus, the dataset is quite small for seasonality. I've tried to fit the best method by using grid search on the p and q hyperparameter : # setting initial values and some bounds for them ps = range(1, 6) d=1 qs = range(1, 6) # creating list with all the possible combinations of parameters parameters = product(ps, qs) Is there anything else I could try? $\endgroup$ – Aastha Jha Aug 17 '20 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.