Difference between cross validation vs model accuracy measures I have a time series ARIMA model and I want to validate the accuracy my prediction. But I dont understand the difference of using cross validation vs model accuracy measures such as MAPE, MAE, MSE and RMSE which can easily generated using R. If they have different purposes, should I be using both?
Could someone help clarify the difference and my understanding?
Thanks!
 A: Cross validation and model accuracy measures are used together to assess and measure prediction accuracy.
Consider a setup where you have a dataset with 1000 observations of a timeseries. Using the whole dataset to "train" your model. Using the model, you can then generate predictions and "test" their accuracy by calculating error measures such as MSE, MAE etc. with respect to the original observations. In this case, the "training dataset" and the "test dataset" are the same. Since training and testing occurs in the same sample, minimising the error measures optimises the in-sample accuracy.
However, only optimising the in-sample accuracy might lead to overfitting of the model on the training dataset, and your prediction of observations from other datasets might be bad. To optimise this out-of-sample accuracy, one usually splits the dataset so that one part is used for training and another one exclusively for testing. You could e.g. define the training set as the first 800 observations and the test set as the last 200 observations.
Cross validation takes this one step further, by creating multiple such training/test splits of your data, calculating prediction errors for each and then minimising the the average prediction errors. For cross-sectional data, you can simply draw 800 random observations and declare them as test data and use the remaining 200 as test. If you do this $K$ times, you have K-fold cross validation.
For time series data this is not so simple, since random sampling destroys the time-series structure of the data. But there are also methods for time-series cross validation.
