9
$\begingroup$

I understand the basic concept of ARCH/GARCH models and the basic mathics behind it. That is, one models the "volatility" of a time series, i.e. the residuals of a time series describing model, which in turn allows the forecasting of volatility.

However, how is the volatility forecast evaluated? In a conditional mean setting, one just compares the actual value with the forecasted value. In a conditional volatility setting, what is being compared? The difference between the aforementioned values and the forecasted residual?

In this case, I can theoretically imagine a scenario in which the residual is always being forecasted correctly, in turn implying that the point forecast is always wrong (if the residual is greater zero). This can not be the correct evaluation method, as it is strongly dependent on how biased the point forecast is. But how do we correctly evaluate ARCH/GARCH forecasts?

Improve this question
$\endgroup$
1
9
$\begingroup$

The point of volatility forecasting is to forecast the full predictive density. For instance, you might assume a normal future density with mean zero, and forecast the one free parameter, which happens to be the variance. Or use some nonparametric approach.

The method of choice for evaluating predictive densities is a proper scoring rule. We have a tag. Its tag wiki contains a few pointers to literature.

As an example, I randomly picked the first relevant article in the current issue of the International Journal of Forecasting, which just happened to be "Forecasting volatility with time-varying leverage and volatility of volatility effects" by Catania & Proietti (2020, IJF). They use the continuous ranked probability score (CRPS), which is one very commonly used proper scoring rule.

Improve this answer
$\endgroup$
6
$\begingroup$

Speaking about evaluating volatility forecasts in general (not GARCH in specific), I will mention an alternative to Stephan Kolassa's answer.

One can also study proper scoring rules for statistics or "properties" of distributions; this area is sometimes called elicitation. There, one can ask the following question: Is there a "proper" scoring rule $S(v, y)$ that evaluates a forecast $v$ of the variance of a random variable using a sample $y$? Here the notion of proper should be that expected score is maximized when $v$ is the true variance.

It turns out that the answer is no. However, there is a trick. There is certainly such a scoring rule for the mean, e.g. $S(u, y) = - (u - y)^2$. It follows that there is a scoring rule for the second moment (not centered), e.g. $S(w, y) = - (w - y^2)^2$.

Therefore, to evaluate a forecast of variance in an unbiased way, it suffices in this case to query the forecast for just two parameters, the first and second moments, which determine the variance. In other words, it's not actually necessary to produce and evaluate the full distribution. (This is basically your proposal: we first evaluate the conditional mean, then the residual, roughly.)

There are of course other measures of volatility than variance, and there is research on whether they are "directly elicitable" (i.e. there exists a proper scoring rule eliciting them) or, if not, their "elicitation complexity" (i.e. how many parameters must be extracted from the underlying distribution in order to evaluate it). One place this is studied is for risk measures in finance. The statistics studied include expectiles, value-at-risk, and conditional-value-at-risk.

There is some general discussion in Gneiting, Making and Evaluating Point Forecasts, Journal of the American Statistical Association (2011). https://arxiv.org/abs/0912.0902 . Elicitation complexity is studied in Frongillo and Kash, Vector Valued Property Elicitation, Conference on Learning Theory (COLT, 2015). http://proceedings.mlr.press/v40/Frongillo15.html

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer. Could u specificy what the variables you are using stand for? $\endgroup$ – shenflow Oct 18 '20 at 13:28
  • 2
    $\begingroup$ The first argument of a scoring rule is a forecast, and the second is a sample (e.g. from the conditional distribution). In my examples, $u$ is our forecast of the mean (so ideally equals the conditional mean), and $w$ is our forecast of the second moment (so ideally equals $\mathbb{E}[Y^2 | X]$). $\endgroup$ – usul Oct 18 '20 at 13:54
1
$\begingroup$

Maybe keep the thing as simple as possible is what shenflow looking for. So:

But how do we correctly evaluate ARCH/GARCH forecasts?

The things is not so different than in conditional mean case, like ARMA. The trick is that you have to care about what you try to forecast.

For example with financial returns ($r_t$) is common to identify the volatility as the squared returns, say $r_t^2$. Now, model like ARCH/GARCH give you a specification for conditional variance: $V[r_t|r_{t-1},…, r_{t-p}]$

For example in ARCH(1) case we have $V[r_t|r_{t-1}]= \omega + \alpha_1 r_{t-1}^2 $

Then for evaluate the forecast accuracy you have to compare the conditional variance (volatility forecast) against the squared return (observed volatility). Then, for accuracy evaluation mean square loss is common. In the ARCH(1) case:

$ MSE [r_t^2 - (\omega + \alpha_1 r_{t-1}^2)] $ for some $t$

Note that behind this example there is the assumption of zero conditional mean for $r_t$. Otherwise, even if the idea is not so different, the second moments and variances do not coincide and the things become more complicated.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Since squared return is a very noisy proxy of volatility (under reasonable assumptions, the upper bound of $R^2$ is around 0.3, if I remember correctly), I am not sure this is a good idea. $\endgroup$ – Richard Hardy Oct 18 '20 at 19:53
  • $\begingroup$ Estimate some ARCH/GARCH models on squared financial returns is common. Them, in general, show notable dependencies in ACF/PACF, while the return series not so much, this is the practical justification for such models. Squared returns can be noisy, it depend on the time frequency also, but consider them as volatility proxy is the standard in model like GARCH. More sophisticated model exist, and, in general, some definition of realized volatility can work better. However, from the question, it seems me that just what I said is what the asker looking for. I can go wrong. $\endgroup$ – markowitz Oct 19 '20 at 12:30
  • $\begingroup$ Note that I imposed here the assumpion of zero conditional mean. If it do not hold, the GARCH model must be write on squared residuals of the model for the mean. However this fact can put in the confusion at first glance. $\endgroup$ – markowitz Oct 19 '20 at 12:30
  • $\begingroup$ I do not oppose anything what you just commented, but I am not persuaded this addresses my concern. Another and perhaps more important question is whether what you suggest amounts to a proper scoring rule. If not, there will be an incentive to estimate/predict something else than conditional variance. $\endgroup$ – Richard Hardy Oct 19 '20 at 12:35
  • 1
    $\begingroup$ I see your point, but the asker write: “In a conditional mean setting, one just compares the actual value with the forecasted value … In a conditional volatility [basic ARCH/GARCH] setting, what is being compared?”. I simply replied to this question, I do not sustain than square loss is the best for volatility accuracy. Or that squared returns (or residuals) are the best for volatility proxy. $\endgroup$ – markowitz Oct 19 '20 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.