1
$\begingroup$

We have four months of data (10 minute interval), this seems have nice pattern (at least for eye ball).

We are using STL to decompose the time series and apply "random walk" to project next month worth of data. Some how projected data is not following the input data pattern. Here is screenshot. The one marked with RED line is forecated output.

We couldn't figure out what behavior in input data might be causing this. Any help would be appreciated.

enter image description here

It is interesting that R also generating similar chart: enter image description here

EDIT: I couldn't upload file, but here is link for data set.

EDIT As everyone commented, it seems there is steep increase in trend at the end of time series. Here is trend chart (trend values extracted by STL): enter image description here

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ The forecast for random walk is constant equal to the last observed value. Thus if you had just a random walk, the forecast would be a straight horizontal line. You have a seasonal component, too, and that seems to be forecast nicely (the pattern resembles the in-sample data). The problem seems to be that the forecast level is off. That could be either if the last observed value was an outlier which drove the random walk forecast off or because there is some programming mistake. Without more details it is hard to tell what is exactly the case. $\endgroup$ – Richard Hardy Mar 3 '15 at 17:36
  • $\begingroup$ Thanks @RichardHardy "constant equal to the last observed value" (or) equal to last observed value of "trend+reminder" from STL? Could you please clarify this? AFAIK, STL breaks observed TS into season, trend and reminder. $\endgroup$ – kosa Mar 3 '15 at 17:54
  • 1
    $\begingroup$ You only mentioned seasonality and random walk, so I based my answer on that. What is random walk in your case? Is it "remainder" or "remainder+trend"? If "remainder" is random walk and there is "trend" extra to it, that could explain why the ultimate forecast seems to be slightly moving downward - that could be due to trend. So please give more details. $\endgroup$ – Richard Hardy Mar 3 '15 at 18:00
  • 1
    $\begingroup$ Then your forecast should be a constant plus the seasonal component, as I noted above. There is nothing I could add to my first comment with regards to why the forecast fails. $\endgroup$ – Richard Hardy Mar 3 '15 at 18:07
  • 1
    $\begingroup$ Where is the random walk? Random walk wanders all around your signal has compact bounds. $\endgroup$ – Cagdas Ozgenc Mar 3 '15 at 20:46
1
$\begingroup$

In the future please provide a reproducible example. As others have pointed out, randomwalk forecast is nothing but the last value of the observed series.So if your deseasonalized data ends at say value 15, your forecast will be value 15 for level/trend. Then you would add seasonal component of had decomposed using STL.

The only way I can think of where you have such as dramatic shift in forecast is if you have a level shift at the end of the series which STL did not capture.

Following is an illustration of my point. I used following code in R for STL and random walk forecast

Lets first consider STL decomposition.

stl(AirPassengers,s.window=7)

Last value of decompostion is as follows: 

          seasonal    trend    remainder
Dec 1960 -51.52714133 495.9921 -12.46495550

So your forecast for trend in STL+random walk would be $trend + remainder$ = 495.9921 - 12.46495550 = 483.5271.

So your future forecast should have a trend value close to ~483. Lets check it:

library("forecast")
stl((stlf(AirPassengers,forecastfunction=rwf,h=36))$mean,s.window=7)

See below for the first few values of the STL decomposed value of your forecast:

Components
           seasonal    trend     remainder
Jan 1961 -38.127116 483.5738  1.705303e-13
Feb 1961 -60.188237 483.5738  1.136868e-13
Mar 1961 -16.891877 483.5738  5.684342e-14
Apr 1961 -16.570300 483.5738  0.000000e+00

As we predicted, the random walk forecast trend is close to 483. enter image description here

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for your time and answer! I have added a link to data set. $\endgroup$ – kosa Mar 3 '15 at 23:16
  • $\begingroup$ As I experiment with this data set, I feel that "trend/level" is the culprit, but I couldn't really find out best solution to handle this (I am Ok to move out from STL + Random walk, if other solution answers this issue). $\endgroup$ – kosa Mar 3 '15 at 23:18
  • $\begingroup$ "if you have a level shift at the end of the series" it seems you are correct here, when I plot trend series extracted from STL, I see steep shift, updated question with screenshot. Does smoothing input series or something like that could help in this case? $\endgroup$ – kosa Mar 4 '15 at 16:13
  • 1
    $\begingroup$ No, you need to a dummy coding to control for steep shift. I'll try to post something if i find time today. $\endgroup$ – forecaster Mar 4 '15 at 16:36
  • 1
    $\begingroup$ Add a dummy code (0 before level shift, 1 after level shift) variable in the xreg statement. You cannot do random walk any more, it has to be ARIMA. $\endgroup$ – forecaster Mar 6 '15 at 19:49
2
$\begingroup$

From the image it seems possible that the one large outlier in the middle may have disproportionately affected the estimate for the drift of the process (its "long-run mean"), and this is why the forecast has been shifted upwards.

As an illustration, assume that the de-seasonalized data have an estimated long-run mean

$$\hat a=\frac 1T\sum_{t=1}^{T}x_i$$

Assume for simplicity that the last element of the sum is disproportionately large (as is the middle one in the image of the question).

Decompose the long-run mean as

$$\hat a=\frac 1T\sum_{t=1}^{T-1}x_i + \frac{x_T}{T}$$

Consider the relative magnitude of the last term

$$\frac {x_t/T}{\frac 1T\sum_{t=1}^{T-1}x_i} = \frac {x_T}{\sum_{t=1}^{T-1}x_i}$$

If this is "large", expect $\hat a$ to be visibly affected by this one observation. In turn $\hat a$ will be used for prediction, shifting the long-term drift of the process.

So I would suggest to remove this on observation from your sample (or dump it artificially) to see what happens.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks for your answer! I am working on dummy that large outlier. Will update soon. $\endgroup$ – kosa Mar 3 '15 at 18:41
  • $\begingroup$ I stubbed dummy value to ZERO for these outliers, still chart looks same. It seems "trend" factor from the STL decomposed causing this forecast to drift little bit. I am not sure why STL picking up upward trend in input series. $\endgroup$ – kosa Mar 3 '15 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.