STL + Random walk failing

Question

We have four months of data (10 minute interval), this seems have nice pattern (at least for eye ball).

We are using STL to decompose the time series and apply "random walk" to project next month worth of data. Some how projected data is not following the input data pattern. Here is screenshot. The one marked with RED line is forecated output.

We couldn't figure out what behavior in input data might be causing this. Any help would be appreciated.

It is interesting that R also generating similar chart:

EDIT: I couldn't upload file, but here is link for data set.

EDIT As everyone commented, it seems there is steep increase in trend at the end of time series. Here is trend chart (trend values extracted by STL):

Answer 1

In the future please provide a reproducible example. As others have pointed out, randomwalk forecast is nothing but the last value of the observed series.So if your deseasonalized data ends at say value 15, your forecast will be value 15 for level/trend. Then you would add seasonal component of had decomposed using STL.

The only way I can think of where you have such as dramatic shift in forecast is if you have a level shift at the end of the series which STL did not capture.

Following is an illustration of my point. I used following code in R for STL and random walk forecast

Lets first consider STL decomposition.

stl(AirPassengers,s.window=7)

Last value of decompostion is as follows:

          seasonal    trend    remainder
Dec 1960 -51.52714133 495.9921 -12.46495550

So your forecast for trend in STL+random walk would be $trend + remainder$ = 495.9921 - 12.46495550 = 483.5271.

So your future forecast should have a trend value close to ~483. Lets check it:

library("forecast")
stl((stlf(AirPassengers,forecastfunction=rwf,h=36))$mean,s.window=7)

See below for the first few values of the STL decomposed value of your forecast:

Components
           seasonal    trend     remainder
Jan 1961 -38.127116 483.5738  1.705303e-13
Feb 1961 -60.188237 483.5738  1.136868e-13
Mar 1961 -16.891877 483.5738  5.684342e-14
Apr 1961 -16.570300 483.5738  0.000000e+00

As we predicted, the random walk forecast trend is close to 483.

Answer 2

From the image it seems possible that the one large outlier in the middle may have disproportionately affected the estimate for the drift of the process (its "long-run mean"), and this is why the forecast has been shifted upwards.

As an illustration, assume that the de-seasonalized data have an estimated long-run mean

$$\hat a=\frac 1T\sum_{t=1}^{T}x_i$$

Assume for simplicity that the last element of the sum is disproportionately large (as is the middle one in the image of the question).

Decompose the long-run mean as

$$\hat a=\frac 1T\sum_{t=1}^{T-1}x_i + \frac{x_T}{T}$$

Consider the relative magnitude of the last term

$$\frac {x_t/T}{\frac 1T\sum_{t=1}^{T-1}x_i} = \frac {x_T}{\sum_{t=1}^{T-1}x_i}$$

If this is "large", expect $\hat a$ to be visibly affected by this one observation. In turn $\hat a$ will be used for prediction, shifting the long-term drift of the process.

So I would suggest to remove this on observation from your sample (or dump it artificially) to see what happens.