3
$\begingroup$

I have forecasts and actuals for panel data (i.e. time-series cross-sectional data). The forecasts are already generated and provided by some source outside of R. I'd like to evaluate the quality of the forecasts.

Are there standard tools in R that perform various diagnostics on the residuals? By diagnostics I mean tests such as:

  • auto-correlation of residuals across the cross-section
  • auto-correlation of residuals along the time series for a given member
  • tests for fixed effects vs. random effects
  • heteroskedasticity, etc.

Or is the best way to perform these diagnostics to perhaps build a panel model using the forecast as the predictor in the panel model?

$\endgroup$
  • $\begingroup$ I don't get what you mean by 'autocorrelation of residuals across the cross-section'. Can you clarify? $\endgroup$ – Vishal Belsare Sep 16 '12 at 11:15
2
$\begingroup$

If by 'auto-correlation of residuals across the cross-section' you mean cross-correlation of residual time series, then the 'ccf' function in R can do cross-correlation between two univariate time series. Related to this look at : Cross-correlation significance in R

Ljung-Box test is implemented in R, and can tell you something the nature of the univariate residuals time series.

(Halbert) White's test helps with testing for heteroscedasticity in residuals.

HTH.

$\endgroup$
1
$\begingroup$

To evaluate the quality of the forecasts, I would first evaluate the quality of the residuals , as you suggested. Take the residuals from each panel separtely into an analytical engine that would test for any ARIMA structure , any anomolous values suggesting pulses,seasonal pulses, level shifts and/or local time trends, any change in model paramters over time and any changes in variance suggesting heterocedasticity. Note that heteroscedasticity can be rectified by weighted least squares, power transformations or Garch . If the analytical engine reports none of these violations , you should be good to go, otherwise the residuals are flawed and your forecasts are subsequently flawed.

$\endgroup$
  • $\begingroup$ +1 Very nice methodological answer. However, The spirit of my was question is looking more for a package in R to handle some or all of these steps. I thank you for your contribution $\endgroup$ – Ram Ahluwalia Sep 18 '12 at 15:43
  • $\begingroup$ @Quant Guy . If you want to post a set of residuals, I would be glad to run it through an analytical engine that I use and show you the results. In this way you can establish specific objectives for your R solution. $\endgroup$ – IrishStat Sep 18 '12 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.