Accuracy of Volatility Forecast

Question

I understand the basic concept of ARCH/GARCH models and the basic mathics behind it. That is, one models the "volatility" of a time series, i.e. the residuals of a time series describing model, which in turn allows the forecasting of volatility.

However, how is the volatility forecast evaluated? In a conditional mean setting, one just compares the actual value with the forecasted value. In a conditional volatility setting, what is being compared? The difference between the aforementioned values and the forecasted residual?

In this case, I can theoretically imagine a scenario in which the residual is always being forecasted correctly, in turn implying that the point forecast is always wrong (if the residual is greater zero). This can not be the correct evaluation method, as it is strongly dependent on how biased the point forecast is. But how do we correctly evaluate ARCH/GARCH forecasts?

Answer 1

The point of volatility forecasting is to forecast the full predictive density. For instance, you might assume a normal future density with mean zero, and forecast the one free parameter, which happens to be the variance. Or use some nonparametric approach.

The method of choice for evaluating predictive densities is a proper scoring rule. We have a scoring-rules tag. Its tag wiki contains a few pointers to literature.

As an example, I randomly picked the first relevant article in the current issue of the International Journal of Forecasting, which just happened to be "Forecasting volatility with time-varying leverage and volatility of volatility effects" by Catania & Proietti (2020, IJF). They use the continuous ranked probability score (CRPS), which is one very commonly used proper scoring rule.

Answer 2

Speaking about evaluating volatility forecasts in general (not GARCH in specific), I will mention an alternative to Stephan Kolassa's answer.

One can also study proper scoring rules for statistics or "properties" of distributions; this area is sometimes called elicitation. There, one can ask the following question: Is there a "proper" scoring rule $S(v, y)$ that evaluates a forecast $v$ of the variance of a random variable using a sample $y$? Here the notion of proper should be that expected score is maximized when $v$ is the true variance.

It turns out that the answer is no. However, there is a trick. There is certainly such a scoring rule for the mean, e.g. $S(u, y) = - (u - y)^2$. It follows that there is a scoring rule for the second moment (not centered), e.g. $S(w, y) = - (w - y^2)^2$.

Therefore, to evaluate a forecast of variance in an unbiased way, it suffices in this case to query the forecast for just two parameters, the first and second moments, which determine the variance. In other words, it's not actually necessary to produce and evaluate the full distribution. (This is basically your proposal: we first evaluate the conditional mean, then the residual, roughly.)

There are of course other measures of volatility than variance, and there is research on whether they are "directly elicitable" (i.e. there exists a proper scoring rule eliciting them) or, if not, their "elicitation complexity" (i.e. how many parameters must be extracted from the underlying distribution in order to evaluate it). One place this is studied is for risk measures in finance. The statistics studied include expectiles, value-at-risk, and conditional-value-at-risk.

There is some general discussion in Gneiting, Making and Evaluating Point Forecasts, Journal of the American Statistical Association (2011). https://arxiv.org/abs/0912.0902 . Elicitation complexity is studied in Frongillo and Kash, Vector Valued Property Elicitation, Conference on Learning Theory (COLT, 2015). http://proceedings.mlr.press/v40/Frongillo15.html

Answer 3

Maybe keep the thing as simple as possible is what shenflow looking for. So:

But how do we correctly evaluate ARCH/GARCH forecasts?

The things is not so different than in conditional mean case, like ARMA. The trick is that you have to care about what you try to forecast.

For example with financial returns ($r_t$) is common to identify the volatility as the squared returns, say $r_t^2$. Now, model like ARCH/GARCH give you a specification for conditional variance: $V[r_t|r_{t-1},…, r_{t-p}]$

For example in ARCH(1) case we have $V[r_t|r_{t-1}]= \omega + \alpha_1 r_{t-1}^2 $

Then for evaluate the forecast accuracy you have to compare the conditional variance (volatility forecast) against the squared return (observed volatility). Then, for accuracy evaluation mean square loss is common. In the ARCH(1) case:

$ MSE [r_t^2 - (\omega + \alpha_1 r_{t-1}^2)] $ for some $t$

Note that behind this example there is the assumption of zero conditional mean for $r_t$. Otherwise, even if the idea is not so different, the second moments and variances do not coincide and the things become more complicated.