Autocorrelation in residuals of a regression model with ARIMA errors (example in Rob Hyndman's book) - Part 1

Question

I am a novice to time series forecasting and I need some help understanding something in Rob Hyndman's excellent Forecasting: Principles and Practice book (3rd edition). After fitting a regression model with ARIMA errors (section 10.3, figure 10.7 for the actual example I am referring to), the book says

"There is clear heteroskedasticity in the residuals, with higher variance in January and February, and lower variance in May. The model also has some significant autocorrelation in the residuals, and the histogram of the residuals shows long tails. All of these issues with the residuals may affect the coverage of the prediction intervals, but the point forecasts should still be ok."

(Bold is mine)

Given that it is important (or a requirement?) for a forecasting model to have residuals with zero mean, and no autocorrelation (mentioned in section 5.4 of the same book), it appears to me that the example in figure 10.7 violates the requirement that there should be no correlations in the residuals of a good forecasting model.

Why does the book then say the point forecasts should still be ok? Does this suggest autocorrelation in residuals will not affect point forecasts? What are the properties of residuals that affects point forecasts?

Part 2 is here

Answer 1

I would say the quoted statement is ambiguous and possibly misleading. Heteroskedasticity does not affect forecasting but serial correlation would make point forecast invalid.

In general, forecast implications of residual diagnostics are:

1. No heteroskedasticity and no serial correlation Forecast can be computed using consistent parameter estimates and forecast/prediction intervals have the right coverage probability.

2. Heteroskedastic but no serial correlation Forecast can be computed using consistent parameter estimates. Forecast/prediction intervals would have the right coverage probability if sample size is large or if robust standard error is used.

3. Serially correlated Parameter estimates are no longer consistent. Forecast and prediction intervals cannot be computed.

For example, take the simplest time series data generating process, the AR(1) model $$ x_t = \rho x_{t-1} + \epsilon_t, $$ and consider the following 3 cases.

Case 1: $\epsilon_t \stackrel{i.i.d.}{\sim} (0, \sigma^2)$

This is the ideal scenario. The residual from fitting the AR(1) model to a sample would not have serial correlation, heteroskedasticity, or thick tails, because the population error term $\epsilon_t$ does not.

The oracle one-period ahead forecast and mean-square forecast error (MSFE) are \begin{align} E[x_{t+1}|x_t] &= \rho x_t,\\ E[ (x_{t+1} - E[x_{t+1}|x_t])^2 ]&= \sigma^2. \end{align}

So to compute one-period ahead forecast based on a sample of size $T$, you simply replace $\rho$ by, say, the OLS/conditional MLE estimate $\hat{\rho}$: $$ x_{T+1 \vert T} = \hat{\rho} x_T. $$ Same for the forecast mean square error $$ \widehat{MSFE}^2 = \frac{1}{T} \hat{\sigma}^2 + \hat{\sigma}^2, $$ where $\hat{\sigma}^2$ is the usual sum of squared residuals divided by $T-1$. The 95% prediction interval is then $x_{T+1 \vert T} \pm 1.96 \times \widehat{MSFE}$. This coverage probability of this prediction interval approaches the nominal coverage probability of 95% in large sample.

($\widehat{MSFE}$ can be computed as follows: \begin{align} \widehat{MSFE}^2 &= E[ (x_{t+1} - \hat{\rho} x_t)^2] \\ &= E[(\hat{\rho} - \rho)^2 x_T^2] + \sigma^2 \\ &\approx \frac{1}{T} \hat{\sigma}^2 + \hat{\sigma}^2. \end{align} In comparison with the oracle MSFE, the first term accounts for estimation error $\hat{\rho} - \rho$. )

Case 2: $(\epsilon_t)$ is (conditionally) heteroskedastic but serially uncorrelated

(For example, $( \epsilon_t )$ could follow an ARCH process. The consistency of $\hat{\rho}$ holds beyond such parametric specifications.)

The residuals from fitting the AR(1) model to a sample would show heteroskedasticity but no serial correlation. The estimate $\hat{\rho}$ is still consistent, and the one-period ahead forecast is still $\hat{\rho} x_T$. A prediction interval of the form $\hat{\rho} x_T \pm \cdots$ would still be correctly centered.

For the mean square forecast error, $$ E[(\hat{\rho} - \rho)^2 x_T^2] \approx \frac{1}{T} \hat{\sigma}^2 $$ is no longer a good approximation. $\hat{\sigma}$ should be replaced by a heteroskedascitity-robust standard error. However, if $T$ is large, this term is negligible, and $$ \hat{\rho} x_T \pm 1.96 \times \hat{\sigma} $$ would still have asymptotic coverage probability of 95%.

Case 3: $(\epsilon_t)$ is serially correlated

(For example, $( \epsilon_t )$ could be itself AR(1).)

The residual from fitting the AR(1) model to a sample would have serial correlation. The estimate $\hat{\rho}$ is no longer consistent (you can check this via simple simulation) and $\hat{\rho} x_T$ is no longer a consistent estimator of $E[x_{T+1}|x_T]$.

The minimal condition required for $\hat{\rho}$ to be consistent is $\frac{1}{T} \sum_{t=1}^T E[x_t \epsilon_t] \rightarrow 0$. This would not be satisfied if $(\epsilon_t)$ has serial correlation.

Caveat: Best Forecast vs. Best Linear Forecast

Forecasting can be discussed in terms of the best forecast $E[x_{T+1}|x_T]$, or best linear forecast. The above discussion is in the context of the best forecast $E[x_{T+1}|x_T]$ (conditional mean of $x_{T+1}$ conditional on $x_T$).

In terms of the best linear forecast, the point forecast $\hat{\rho} x_T$ is still valid under Case 3. The difference is that while $\hat{\rho}$ no longer consistently estimates $\rho$, it still captures linear correlation between $x_{T}$ and $x_{T+1}$: $$ \hat{\rho} \stackrel{p}{\rightarrow} \frac{Cov(x_{t+1}, x_t)}{Var(x_t)} \, (\neq \rho). $$ The forecast interval $$ \hat{\rho} x_T \pm 1.96 \times \hat{\sigma}_{HAC} $$ would have the correct asymptotic coverage probability (with respect to the best linear forecast, not the best forecast) if $\hat{\sigma}^2_{HAC}$ is the heteroskedasticity autocorrelation robust (HAC) estimate of long-run variance computed from the residuals.

Answer 2

The short answer is that, usually, autocorrelation does not impact the estimates of coefficients, but impacts the variances. That's why he's saying that point forecasts will not change, but confidences will. Also, in time series regression the residuals are almost always correlated.

In other words in a model $y_t=X_t\beta+\varepsilon_t$, where $\varepsilon_t$ is ARIMA, if you ignore autocorrelation in $\varepsilon_t$, then your $\hat\beta$ are still Ok, but their p-values and variances $\hat\sigma^2_\beta$ can be messed up. Thus the quip on the point forecast $\hat y_{t+h}=X_{t+h}\hat\beta$ being Ok. He was careful to say "should still be ok," making it not absolute statement, but more of a practical advice, with which I agree.

Non zero mean is difficult to detect, because out of regression by construction the residuals will come with zero (unconditional) mean. Non zero mean error is an issue, of course, but it's more subtle than many think. Here's how it's expressed in conditional terms: $E[\varepsilon|X]=0$. One situation that violates this condition is when errors' mean varies with predictors. For instance, you overestimate for large values of predicted $\hat y$ and underestimates for small values. This is why it's recommended to plot residuals vs predicted chart.