21
$\begingroup$

I am trying to learn how to use Neural Networks. I was reading this tutorial.

After fitting a Neural Network on a Time Series using the value at $t$ to predict the value at $t+1$ the author obtains the following plot, where the blue line is the time series, the green is the prediction on train data, red is the prediction on test data (he used a test-train split) p1

and calls it "We can see that the model did a pretty poor job of fitting both the training and the test datasets. It basically predicted the same input value as the output."

Then the author decides to use $t$, $t-1$ and $t-2$ to predict the value at $t+1$. In doing so obtains

p2

and says "Looking at the graph, we can see more structure in the predictions."

My question

Why is the first "poor"? it looks almost perfect to me, it predicts every single change perfectly!

And similarly, why is the second better? Where is the "structure"? To me it seem much poorer than the first one.

In general, when is a prediction on time series good and when is it bad?

$\endgroup$
4
  • 4
    $\begingroup$ As a general comment, most ML methods are for cross-sectional analysis, and need adjustments to be applied for time series. The main reason is autocorrelation in data, while in ML often the data is assumed independent in most popular methods $\endgroup$
    – Aksakal
    Oct 4, 2017 at 16:46
  • 13
    $\begingroup$ It does a great job of predicting every change... right after it happens! $\endgroup$
    – hobbs
    Oct 4, 2017 at 21:19
  • $\begingroup$ @hobbs , I am not trying to use t, t-1, t-2 etc to predict t+1. I was wondering if you know how many terms in the past is best to use. If we use too many, are we overfitting? $\endgroup$ Oct 5, 2017 at 17:54
  • $\begingroup$ It would have been more illuminating to plot the residuals. $\endgroup$
    – reo katoa
    Oct 5, 2017 at 19:59

3 Answers 3

33
$\begingroup$

It's sort of an optical illusion: the eye looks at the graph, and sees that the red and blue graphs are right next to each. The problem is that they are right next to each other horizontally, but what matters is the vertical distance. The eye most easily see the distance between the curves in the two-dimensional space of the Cartesian graph, but what matters is the one-dimensional distance within a particular t value. For example, suppose we had points A1= (10,100), A2 = (10.1, 90), A3 = (9.8,85), P1 = (10.1,100.1), and P2 = (9.8, 88). The eye is naturally going to compare P1 to A1, because that is the closest point, while P2 is going to be compared to A2. Since P1 is closer to A1 than P2 is to A3, P1 is going to look like a better prediction. But when you compare P1 to A1, you're just looking at how well A1 is able to just repeat what it saw earlier; with respect to A1, P1 isn't a prediction. The proper comparison is between P1 v. A2, and P2 v. A3, and in this comparison P2 is better than P1. It would have been clearer if, in addition to plotting y_actual and y_pred against t, there had been graphs of (y_pred-y_actual) against t.

$\endgroup$
2
  • 2
    $\begingroup$ This is the better answer as the other one does not even mention why a "good-looking" forecast is actually poor, while you do a great job at that! $\endgroup$ Oct 5, 2017 at 5:17
  • 1
    $\begingroup$ +1 I completely support plotting the residuals, $y_t - \hat y_t$, as that will show the vertical distance. A scatterplot of $y_t$ and $\hat y_t$ could be useful, too, especially with the line $y_t = \hat y_t$ under those point to show a perfect fit, though this will lose the relationship to time. $\endgroup$
    – Dave
    Nov 3, 2023 at 1:51
22
$\begingroup$

Why is the first "poor"? it looks almost perfect to me, it predicts every single change perfectly!

It is a so-called "shifted" forecast. If you look more closely at chart 1, you see that the prediction power is only in copying almost exactly the last seen value. That means model learned nothing better, and it treats the time series as a random walk. I guess that the problem may be in the fact you use the raw data that you feed to the neural network. These data are non-stationary which causes all the trouble.

$\endgroup$
6
  • 10
    $\begingroup$ In forecasting this is called "naive" forecast, i.e. use the last observed as a forecast $\endgroup$
    – Aksakal
    Oct 4, 2017 at 16:44
  • $\begingroup$ Thank you! @Aksakal do you know how many previous values should be used for prediction? $\endgroup$ Oct 5, 2017 at 17:55
  • $\begingroup$ Focus on stationarity. A couple of stationary lags should be quite good for this time series. Better than 100 nonstationary lags. $\endgroup$ Oct 5, 2017 at 17:58
  • $\begingroup$ in time series there's a way to get a good guess on lag structure through ACF and PACF, lookup this forum, there were many posts on how it's done $\endgroup$
    – Aksakal
    Oct 5, 2017 at 18:17
  • $\begingroup$ @AlexeyBurnakov so does it mean I should transform it to be stationary? $\endgroup$ Oct 5, 2017 at 19:45
0
$\begingroup$

This is a naive "copy from last" prediction which is essentially not informative as a forecasting model. One practical technique (not sure about theoretical soundness) is to train the model by making multi-step ahead predictions. This seems to help in practice which in a way forces the model to learn more "persistent patterns".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.