Forecasting an index with google in R

Question

I am trying to predict an index using Google Trend Data. I try to orientate myself by this paper. In this paper the authors use the three variables: Sales, Index and SearchFrequency to forecast the Index(he also forecast the sales but in my case its not necessary). Already here a question arise for me:

(1) In this paper the authors use a "seasonal autoregressive (AR) model" for multivariate data. I thought that AR models can only be used for univariate data. So I am wondering if they used an AR model for every time series separately and put the outputs into a linear regression model? I feel stupid asking that question but that's how I can explain it to myself.

Back to my case: These are my two time series:

Search Frequency:freq <- ts(frequency, start=2004, frequency = 4)(blue-line)

Index:index <- ts(index, start=2004, frequency = 4) (red-line)

Both time series are quarterly data points. As you can see in the graph of "SearchFrequency" there is a slight seasonality in the fourth quarter. Because both time series are non-stationary my first step was to transfer them into stationary time series.

(2) Do both time series have to be stationary?

I used autoplot(diff(freq)) and autoplot(diff(diff(index)))

(3) sometimes I also read about taking the log instead of differencing. After log however my time series still weren't stationary so I decided to take the differencing, but it would be still interesting to know when to use what.

after differencing the time series look like this:

Search Frequency: Index:

Search frequency for me seems to be stationary with high variance now and index seems also to be stationary but I had to difference twice because after the first time there was still a trend.

To determine appropriate values for p,q I used the ACF and PACF plot:

For Index: (4) Looking at the ACF Plot we see two significant spikes and in PACF we see three spikes. does that mean that I should use p=2 and q=3?

For Frequency:

(5) Looking at the ACF and PACF plots of Frequency its getting really confusing for me. The ACF shows a sinusoidal pattern with a lot of spikes and the PACF has 5 spikes if we ignore the spike at lag 7. Which information can I get out of this?

(6) what is the next step? Should I do the auto.arima() with both time series and try to compare them to arima() with the p,d,q I got out of the ACF and PACF plots, compare them and choose the model which has the lowest AIC value?

I am really grateful for any kind of help. I worked with "forecasting. principles and practice" by Rob Hyndman which helped me already a lot.

EDIT:

  frequency:
3
2
5
3
3
4
6
5
8
6
5
4
4
7
7
5
9
8
9
8
9
11
11
11
15
16
19
12
26
30
29
24
32
36
38
28
39
45
48
39
52
55
58
44
65
68
69
59
70
71
75
58
77
79
77
71
90
88
94
75
98
90

index:
83.9
82.8
81.9
80.6
84.2
82.7
84.4
82
83.2
83.5
82.3
83
80
81.7
81.6
81.6
82.8
83.1
81.6
81.8
81.8
83.1
82.9
84.2
83
84.3
84.3
83.8
86
87.1
86.7
87.3
87.9
89.1
90.4
91.6
91.3
93.1
92.9
93
93.8
95.7
96.2
96.2
97.8
99.9
100.4
101.8
103.9
106.9
108.8
110.4
110.9
113.1
115
117.3
118.3
120.6
123.1
124.6
124.4
126.9

Answer 1

Please read https://autobox.com/pdfs/regvsbox-old.pdf to get an idea of causal analysis in time series. Essentially one pre-whitens the data in order to be able to use standard methodologies to IDENTIFY a possible model.

http://www.math.cts.nthu.edu.tw/download.php?filename=569_fe0ff1a2.pdf&dir=publish&title=Ruey+S.+Tsay-Lec1 will also be of some help.

If you post your data in a csv vile I will try and help further.

EDIT AFTER RECEIPT OF DATA:

I took 62 quarterly values fro the Y variable (freq) and the X variable INDEX into AUTOBOX , a piece of software that I have helped to develop, It treats the problem in an holistic manner incorporating differencing when necessary , error variance components to stabilize the model's residual variance , outliers/pulses/season pulses as needed. Sometimes software has features that are unneeded for a particular problem ... sometimes (this case) the data requires a sequence of complex modelling steps that are non-trivial. Your particular data set is non-trivial There is an R version of AUTOBOX .

Here is the plot of the two series visually suggesting some linkage. Mode identification required using a filter on both Y and X of the following form yielding cross-correlations that can be interpreted to suggest the form of the Transfer Function model.

Here is the first iteration .

The residuals from this model suggested possible improvements incorporating an ar(1) augmentation and 2 seasonal pulses suggesting latent deterministic seasonal.

An examination of the model's residuals suggested a deterministic change point in the model's error variance using the TSAY test for constant error variance here [] . While it is true that the model error variance is larger at higher levels of Y (suggesting the need for a log transform of Y), it is more trueer that the model's error variance changed/increased deterministically between time region 1-26 versus time region 27-64 4. This is visually obvious when one looks at a plot of Y by itself

The final model is here with acf/pacf of the model's residuals here

The forecast for Y depends on forecasts for X which are shown here based upon the AROMA model for X

leading to a forecast table for Y and an accompanying graph

The Actual/Fit and Forecast graph is here