15
$\begingroup$

I am trying to learn how to use Neural Networks. I was reading this tutorial.

After fitting a Neural Network on a Time Series using the value at $t$ to predict the value at $t+1$ the author obtains the following plot, where the blue line is the time series, the green is the prediction on train data, red is the prediction on test data (he used a test-train split) p1

and calls it "We can see that the model did a pretty poor job of fitting both the training and the test datasets. It basically predicted the same input value as the output."

Then the author decides to use $t$, $t-1$ and $t-2$ to predict the value at $t+1$. In doing so obtains

p2

and says "Looking at the graph, we can see more structure in the predictions."

My question

Why is the first "poor"? it looks almost perfect to me, it predicts every single change perfectly!

And similarly, why is the second better? Where is the "structure"? To me it seem much poorer than the first one.

In general, when is a prediction on time series good and when is it bad?

Improve this question
$\endgroup$
  • 3
    $\begingroup$ As a general comment, most ML methods are for cross-sectional analysis, and need adjustments to be applied for time series. The main reason is autocorrelation in data, while in ML often the data is assumed independent in most popular methods $\endgroup$ – Aksakal Oct 4 '17 at 16:46
  • 11
    $\begingroup$ It does a great job of predicting every change... right after it happens! $\endgroup$ – hobbs Oct 4 '17 at 21:19
  • $\begingroup$ @hobbs , I am not trying to use t, t-1, t-2 etc to predict t+1. I was wondering if you know how many terms in the past is best to use. If we use too many, are we overfitting? $\endgroup$ – Euler_Salter Oct 5 '17 at 17:54
  • $\begingroup$ It would have been more illuminating to plot the residuals. $\endgroup$ – reo katoa Oct 5 '17 at 19:59
23
$\begingroup$

It's sort of an optical illusion: the eye looks at the graph, and sees that the red and blue graphs are right next to each. The problem is that they are right next to each other horizontally, but what matters is the vertical distance. The eye most easily see the distance between the curves in the two-dimensional space of the Cartesian graph, but what matters is the one-dimensional distance within a particular t value. For example, suppose we had points A1= (10,100), A2 = (10.1, 90), A3 = (9.8,85), P1 = (10.1,100.1), and P2 = (9.8, 88). The eye is naturally going to compare P1 to A1, because that is the closest point, while P2 is going to be compared to A2. Since P1 is closer to A1 than P2 is to A3, P1 is going to look like a better prediction. But when you compare P1 to A1, you're just looking at how well A1 is able to just repeat what it saw earlier; with respect to A1, P1 isn't a prediction. The proper comparison is between P1 v. A2, and P2 v. A3, and in this comparison P2 is better than P1. It would have been clearer if, in addition to plotting y_actual and y_pred against t, there had been graphs of (y_pred-y_actual) against t.

Improve this answer
$\endgroup$
  • 2
    $\begingroup$ This is the better answer as the other one does not even mention why a "good-looking" forecast is actually poor, while you do a great job at that! $\endgroup$ – Richard Hardy Oct 5 '17 at 5:17
17
$\begingroup$

Why is the first "poor"? it looks almost perfect to me, it predicts every single change perfectly!

It is a so-called "shifted" forecast. If you look more closely at chart 1, you see that the prediction power is only in copying almost exactly the last seen value. That means model learned nothing better, and it treats the time series as a random walk. I guess that the problem may be in the fact you use the raw data that you feed to the neural network. These data are non-stationary which causes all the trouble.

Improve this answer
$\endgroup$
  • 8
    $\begingroup$ In forecasting this is called "naive" forecast, i.e. use the last observed as a forecast $\endgroup$ – Aksakal Oct 4 '17 at 16:44
  • $\begingroup$ Thank you! @Aksakal do you know how many previous values should be used for prediction? $\endgroup$ – Euler_Salter Oct 5 '17 at 17:55
  • $\begingroup$ Focus on stationarity. A couple of stationary lags should be quite good for this time series. Better than 100 nonstationary lags. $\endgroup$ – Alexey Burnakov Oct 5 '17 at 17:58
  • $\begingroup$ in time series there's a way to get a good guess on lag structure through ACF and PACF, lookup this forum, there were many posts on how it's done $\endgroup$ – Aksakal Oct 5 '17 at 18:17
  • $\begingroup$ @AlexeyBurnakov so does it mean I should transform it to be stationary? $\endgroup$ – Euler_Salter Oct 5 '17 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.