How can I be confident about my forecasts and improve my methodologies?

Question

Background

I usually do a fair amount of forecasting using ARIMA, linear or multivariate regressions, polynomial trends, etc. A lot of this forecasting is for simplistic use and not really basis for any decision making. I'm getting into forecasting and "advanced statistics" in general and would like to excel at forecasting and trending especially for engineering problems.

I've done a few classes at university way back when: Statistics 1 for engineers, statistical systems (covered, anova, manova, time series, chai-square, and TSP. I also did a specific class on validation and calibration of engineering models (didn't cover much statistics, but was relevant).

What I do

1. When looking at a time series data set, I usually plot it first, check for any seasonality, outliers, any general trends.
2. I then try to fit the model to a few regressions and chose the best one based on RMSE or MAE, and looking at the residual plots.
3. I then manipulate the initial data by either log(data) data^k(time)
4. I then recheck to see how my data looks like and how it's fitting

I have an average idea on fitting models. For example, when doing an ARIMA mode, I'll be looking at the acf, pacf and identify lags to produce the appropriate constants. Although, this could be improved as I usually have to refer back to my notes and I don't understand the whole acf/pacf process 100% yet. (I'm not fluent yet at this).

Problem

One thing is, I'm never confident about my forecasts. Yes, visually the new forecast squiggly line matches my expectations. I usually add 2 lines +/- 2*sigma. But still, how can I say, I dassouki, approve of this forecast to be reflective of the prior years data. However, let's say I change the constants in an ARIMA a bit, or ask the model to backcast or analyze seasonal vs. non seasonal data, I get really confused and I get to a stage where I don't know what test for, and when I test for normality, accuracy, or validation, am I reading the output correctly.

Question

• How can I be confident about my forecasts, I know that all modeling is "false"; however, How can I be certain that the assumptions I made, factors I used, and results I produced will somewhat reflect the "future"?

Answer 1

You can be confident of your models if you cross-validate them! You can do by "rolling" through your dataset one observation at a time, forecasting 1 step ahead. At the end of the process compare your 1-step forecasts to the actual values to get some idea of how your model performs out-of-sample.

You can find this process explained in more detail at the end of this post. You can find R code for cross-validating time series here, here and here.