0
$\begingroup$

I am using packages {rugarch} for forecasting and {forecast} for Diebold - Mariano test. As a first step, I am specifying the first AR-GARCH model for financial time series (AAPL Nasdaq) using ugarchspec{rugarch}

   spec1 <- ugarchspec(
  variance.model =
    list(model = "sGARCH", garchOrder = c(1,1), submodel = NULL, external.regressors = NULL, variance.targeting = FALSE),
  mean.model =
    list(armaOrder = c(2, 0), external.regressors = NULL), distribution.model = "norm" )

Then I fit it using ugarchfit{rugarch}

modelfit1=ugarchfit(spec1, data=AAPL_returns, out.sample = 200)
modelfit1

Then I forecast it using ugarchforecast{rugarch}

modelfor1=ugarchforecast(modelfit1, n.ahead = 1, n.roll
                        = 200, out.sample = 200)

This is my first forecast to be compared in Diebold-Mariano test. The other forecast is very similar to the first one but the AR-GARCH model is enhanced by the external regressor, which is data (GGL) from Google Trends on "AAPL" keyword:

spec2 <- ugarchspec(
      variance.model =
        list(model = "sGARCH", garchOrder = c(1,1), submodel = NULL, external.regressors = GGL, variance.targeting = FALSE),
      mean.model =
        list(armaOrder = c(2, 0), external.regressors = NULL), distribution.model = "norm" )

Then I fit it:

modelfit2=ugarchfit(spec2, data=AAPL_returns, out.sample = 200)
modelfit2

And I forecast it:

modelfor2=ugarchforecast(modelfit2, n.ahead = 1, n.roll
                        = 200, out.sample = 200)

So I am ending up with two formal class uGARCHforecasts - modelfor1, modelfor2

Now comes the problematic part. As Diebold-Mariano dm.test{forecast} is working with forecasting errors, I need to save them from my previous forecast which I have done this way:

e1 <-(modelfor1@model$modeldata$residuals)
e2 <-(modelfor2@model$modeldata$residuals)

accuracy(e1)
accuracy(e2)
dm.test(e1,e2,h=1)

The dm.test runs without problem, however the accuracy(e1) is unable to provide information on summary measures of the forecast accuracy, since it returns

Error in accuracy(e1) : Unable to compute forecast accuracy measures

I would like to raise two questions:

1) Is this model as a whole well specified, does it capture a legit comparison of two well specified forecast?

2) What is the problem with inability to return accuracy measures in accuracy(e1) ?

Edit:

I suggest a second model which works with squared residuals as a proxy to forecast in-sample volatility:

modelsigma1 = as.data.frame(modelfit1@fit$sigma)
modelsigma2 = as.data.frame(modelfit2@fit$sigma)
ret.sq = AAPL_returns^2
modelsigma1.sq = modelsigma1^2
modelsigma2.sq = modelsigma2^2
dm.test((ret.sq - modelsigma1.sq), (ret.sq - modelsigma2.sq), alternative = c("two.sided"), h = 1, power = 1)
$\endgroup$
2
  • $\begingroup$ Regarding the failure of accuracy, type in accuracy and press "Enter". You will see the source code. Based on that you can manually check which if clause fails (there are a few of them) for your particular forecast, and then you can think of how to repair that. $\endgroup$ Commented Jun 23, 2016 at 10:35
  • $\begingroup$ I was going through my old answers and noticed this one was not accepted. Do you perhaps need further clarification? $\endgroup$ Commented Feb 15, 2017 at 10:30

1 Answer 1

1
$\begingroup$

These

e1 <-(modelfor1@model$modeldata$residuals)
e2 <-(modelfor2@model$modeldata$residuals)

are not forecast errors. They are in-sample residuals from the conditional mean model.

First, in-sample residuals are not the same as out-of-sample forecast errors.
Second, you care about forecasts of conditional variance, not point forecasts. So even if you had out-of-sample forecast errors of the conditional mean model (the difference between point forecasts and actual realizations), that would not be what you need.

You could get an estimate of realized unconditional variance using high-frequency data, but I don't think you could get it from the data you used for fitting the GARCH model (except perhaps when taking on extra assumptions). I am not sure how to get realized values of conditional variance.

$\endgroup$
2
  • $\begingroup$ Thanks Richard. You are right that volatility is unobservable so it is unclear what we use as observed values. I then suggest to use some kind of proxy. Squared returns are not very accurate but should be satisfactory. The code is then: modelsigma1 = as.data.frame(modelfit1@fit$sigma) modelsigma2 = as.data.frame(modelfit2@fit$sigma) ret.sq = AAPL_returns^2 modelsigma1.sq = modelsigma1^2 modelsigma2.sq = modelsigma2^2 dm.test((ret.sq - modelsigma1.sq), (ret.sq - modelsigma2.sq), alternative = c("two.sided"), h = 1, power = 1) $\endgroup$
    – John
    Commented Jun 25, 2016 at 5:54
  • $\begingroup$ That is better, except that squared returns are a terribly noisy proxy. $\endgroup$ Commented Jun 25, 2016 at 6:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.