Residual analysis of cross-sectional time-series forecasts

Question

I have forecasts and actuals for panel data (i.e. time-series cross-sectional data). The forecasts are already generated and provided by some source outside of R. I'd like to evaluate the quality of the forecasts.

Are there standard tools in R that perform various diagnostics on the residuals? By diagnostics I mean tests such as:

• auto-correlation of residuals across the cross-section
• auto-correlation of residuals along the time series for a given member
• tests for fixed effects vs. random effects
• heteroskedasticity, etc.

Or is the best way to perform these diagnostics to perhaps build a panel model using the forecast as the predictor in the panel model?

Answer 1

If by 'auto-correlation of residuals across the cross-section' you mean cross-correlation of residual time series, then the 'ccf' function in R can do cross-correlation between two univariate time series. Related to this look at : Cross-correlation significance in R

Ljung-Box test is implemented in R, and can tell you something the nature of the univariate residuals time series.

(Halbert) White's test helps with testing for heteroscedasticity in residuals.

HTH.

Answer 2

To evaluate the quality of the forecasts, I would first evaluate the quality of the residuals , as you suggested. Take the residuals from each panel separtely into an analytical engine that would test for any ARIMA structure , any anomolous values suggesting pulses,seasonal pulses, level shifts and/or local time trends, any change in model paramters over time and any changes in variance suggesting heterocedasticity. Note that heteroscedasticity can be rectified by weighted least squares, power transformations or Garch . If the analytical engine reports none of these violations , you should be good to go, otherwise the residuals are flawed and your forecasts are subsequently flawed.