Residuals in a Machine Learning with forecasting

Question

I'm affraid to confuse something. Here's my assumption :

I have a timeseries that I split in 2 parts Learning data and test data.

A residual in forecasting is the difference between an observed value and its forecast based on other observations: ei=yi−y^iei=yi−y^i.

I want to see how good my model is so I want t know the residuals.

But which data are my observations ? For me, it's my learning data + my test data. Because i had prediction on all my data. (On the learning data and the most important on my test data)

EDIT : So i'm calcutating the residuals on all my data.

EDIT 2 : Paper

Am I right ?

Thanks for your help.

Answer 1

I see two steps

1 -- MODEL SELECTION

Here we use a train and a test error to select a model. In this contest, model refers to a the mathematical formulation of a solution (a function $\hat f (x, \theta)$) rather than a specific instance of a model where every parameter ($\theta$) has been set. Or, in other words, the performance of a method of producing a model, not of the model itself (see)

In this step. You will deal with test and train error. That is residuals on the test and train data. The analysis should take into account and discriminate between test and train. You can read about the bias/variance tradeoff here.

I would also recommend

• Hastie, Tibshirani, Friedman “Elements of Statistical Learning” 2001
• This very good visual introduction: http://scott.fortmann-roe.com/docs/BiasVariance.html

In a nutshell (source):

2 -- MODEL FITTING

In this phase, after having selected a model, we fit it to the data. In case of a simple linear regression, we can find the $\theta$ that minimizes the cost function.

Here we make use of the whole dataset. We make use of the whole dataset because we have already analyzed how a model generalizes and the risk of overfitting in step 1. Now it is time to estimate $\theta$ and the more data we have, the better it is.

I believe that it is in this context that it is meaningful speaking of prediction and confidence intervals.

I mention here the two because they are often confused.

Although both are centered at $\hat y$, the prediction interval is wider than the confidence interval, for a given $x$ and confidence level. This makes sense, since the prediction interval must take account of the tendency of y to fluctuate from its mean value, while the confidence interval simply needs to account for the uncertainty in estimating the mean value (source)