Using GARCH rolling forecast in R to calculate MAE?

My intention is to calculate the MAE for different (G)ARCH-models (comparing the one-step-ahead forecast for $\sigma$ with the absolute return that day).

The formula for MAE is actually clear, but I'm not quite sure which two series to use, when I do a rolling forecast in R for a (G)ARCH-model including mean.

Some Output I can extract after the roll.forecast is a series of "$\mu$" as well as a series of "$\sigma$". Is it okay to compare this mentioned $\sigma$ with the absReturn [MAE=Sigma-absReturn] or do I have to consider the $\mu$ as well and compare $\sigma$ and (absReturn-$\mu$) [MAE=$\sigma$-(absReturn-$\mu$)]?

Because according to theory return = conditional mean + cond. volatility and if this $\sigma$ is identical to cond. volatility I have to adjust the return?

• Are you trying to evaluate the point forecasts or the volatility forecasts? – Richard Hardy Dec 7 '16 at 12:24
• The volatility forecast. – jürgen Dec 7 '16 at 12:28
• And how do you define MAE for a GARCH model? Is it the mean absolute forecast error for the conditional variance, or is it something else? – Richard Hardy Dec 7 '16 at 12:45
• yes, exactly.... I do a one step forecast for sigma[t+1] with, for example, a GARCH(1,1)-model, and compare this forecastwith the observed absoluteReturn at t+1... by doing this for say 500 days I think I have a good measure of how accurate a particular model is (and I can compare it with EGARCH(1,1) or any other model) – jürgen Dec 7 '16 at 12:59
• The problem is that the true $\sigma_{t}$ is unobservable -- we cannot measure it, so forecast accuracy measures like MAE might not apply directly. You could read a bit about evaluation of volatility or density forecasts and choose a better measure. – Richard Hardy Dec 7 '16 at 13:11

I understand that you want to evaluate volatility forecasts by comparing the forecasted standard deviation of the model error with the realized absolute value of the model error. This can be done by comparing the forecast of sigma from the GARCH output with the absolute difference between the point forecast and the realized value.

But standard deviation is not equal to expected absolute value, so this may not be a good way of evaluating your GARCH forecasts. Alternatives could be,

• use a GARCH-type model for expected absolute deviations rather than variances, and compare with realized absolute deviations;
• evaluate the variance forecasts against squares of errors due to point forecasts (but recall that the latter is a pretty noisy proxy for the former, so do not expect very good results there).

This is not possible. First of all definition of "volatility" is model dependent, second of all it is not observable under GARCH.

Your frustration with the mean and volatility interaction is also the evidence of model dependent volatility. But you are not at fault here. Fault is those who infested the books with the ARMA-GARCH models and constant quadratic variance ideas (which are contradictory by the way). These are just models.

There are different types of volatilities such as quadratic variation, jump type volatility, etc. All these definitions come from underlying models.

You just defined yours as absolute return for example. If you compare this with say GARCH basically you are comparing apples and oranges as the volatility definitions will be different in the first place.

Let's assume for a minute that ARMA-GARCH is a good conditional mean and volatility model for your data. Then you have two possible paths:

• Compare AIC of different candidate models you build
• Compare likelihood of candidate models on data that you haven't used while estimating the models.

By using ARMA-GARCH instead of only GARCH you take into consideration conditional mean effects if there are any (which there is not in financial asset price data as markets are very efficient), and make sure that it doesn't blend with the quadratic variance.

• Many thanks for your comment! May I ask you whether there are any other information-criteria next to AIC which are advisable. I know there is for example the BIC, but is it sufficient to to make a dicision for one model based on just one or two criteria? – jürgen Dec 7 '16 at 15:22
• @stefan There are lots of and inconclusive discussions regarding which information criterion is better. They are all approximations. I am on the AIC camp. But if you don't like them use the second option of comparing likelihoods on unseen data. – Cagdas Ozgenc Dec 7 '16 at 15:27