On forecasting, the mean squared error and realized volatility

Question

Say one has finished estimating a correctly specified GARCH(1,1) on a daily time series and now wants to evaluate the accuracy of the one step ahead forecasts what steps or tests could one do?

I understand that the MSE

$$MSE = \frac{1}{N} RSS = \frac{1}{N} \sum (\hat{\sigma}_i -\sigma_i)^2$$

can be computed where $N$ is the number of samples and $\hat{\sigma}_i$ is the estimated one step ahead volatility. Because we do not know the realized volatility $\sigma_i$ we can use the squared return of that day as proven here.

But is the one step ahead predictor not already defined as the value $\hat{\sigma}$ of the volatility that minimizes the MSE? So why compute this measure if it going to be the minimum across models anyway?

Moreover do I understand correctly that the practical way to compute the $MSE$ for the one step ahead forecast is:

1. Estimate the correctly specified model on your data (returns) except the last data point.
2. Compute the one step ahead forecast of your model.
3. Repeat step 1 but take away an other data point.

Doing this process $N$ times one obtains $N$ $\hat{\sigma}_i$ that can then by utilized in the given formula. Am I getting this right?

Also I understand from this paper (Bollerslev 1998) that utilizing the squared daily return to approximate the realized volatility leads to noise. For example a better estimate of realized daily volatility would be the sum of 30 minutes squared returns of that day. Is this correct?

So to get a "better" $MSE$ I could substitute every $\sigma_i$ with the sum of 30 minutes squared returns of that day instead of simply the daily squared return? Is this how you use realized volatility to evaluate the goodness of your forecasts?

I will end this rambling by asking for a good reference in evaluating the accuracy of the forecasts using realized volatility because it is obvious that I am very confused. I can't seem to find a good text on my own.

Because I have asked a lot of questions in a confused fashion I will put a bounty of 50 reputation so you might want to wait to answer (but you could comment to give me a reference straight away if you happen to know one).

Answer 1

But is the one step ahead predictor not already defined as the value $\hat \sigma$ of the volatility that minimizes the MSE?

If you estimate the GARCH model using maximum likelihood then the fitted values $\hat\sigma_t$ are the likelihood-maximizing values (subject to the GARCH(1,1) functional form) which need not coincide with MSE-minimizing values. That depends on the distribution assumed for the likelihood calculation.

Also, when fitting the model on a data sample indexed from $1$ to $T$, the fitted value $\hat\sigma^2_t$ for $t<T$ utilizes the information not only from $1\leqslant\tau<t$ but also from $t\leqslant\tau\leqslant T$. Meanwhile, to fairly evaluate the forecast performance, you should not allow the estimate to be calculated using future data; instead, you need to check the forecast accuracy out of sample.

So why compute this measure if it going to be the minimum across models anyway?

Because you may want to know what the actual values of MSE is. Knowing that the MSE is minimal does not tell you what its value is. (See also the answer to the previous question.)

Moreover do I understand correctly that the practical way to compute the $MSE$...

That will be the mean squared forecast error. If you want in-sample MSE, just use the fitted values from the model estimated on the whole sample. The former should give an unbiased estimate of model performance, while the latter will be too optimistic in that respect.

Also I understand from this paper (Bollerslev 1998) that utilizing the squared daily return to approximate the realized volatility leads to noise.

Very important point indeed! The response variable of the GARCH model is measured with noise when squared errors are used as proxies; this noise may be quite substantial. When trying to assess model fit, the measurement error associated with the dependent variable may cause quite some trouble.

For example a better estimate of realized daily volatility would be the sum of 30 minutes squared returns of that day.

On the first thought, that could be a valid option. If that was proposed in the Andersen and Bollerslev (1998) paper, then it must be fine.

So to get a "better" MSE I could substitute every σi with the sum of 30 minutes squared returns of that day instead of simply the daily squared return? Is this how you use realized volatility to evaluate the goodness of your forecasts?

That makes sense to me.

I will end this rambling by asking for a good reference in evaluating the accuracy of the forecasts using realized volatility...

Regarding references, I think the Andersen and Bollerslev (1998) paper is quite relevant and complete. Also see Patton & Sheppard "Evaluating volatility and correlation forecasts" (2009) (free version here) and other works by Patton.

Answer 2

As to your estimator of the one-step forecast: it sounds like a "jack-knife"; regarding the general problem, it sounds like one of time series filtering; the Kalman filter is one of the first techniques found in this (huge) topic, and valid in the limit of small time-steps and Gaussian-distributed realizations.

One in general tests a model on the basis of the "evidence" (likelihood of the model parameters times prior over models, integrated over the parameters), as per Bayes' theorem.

More to the point of your question:

So my question is, does it make sense to compute the mean absolute error using the
minimal mean squared error predictor? is this valid for all types of loss functions? It
seems to me there should actually be an optimal predictor for every different loss
function chosen?

The mean absolute error is indeed different from the LS one; the absolute value function is not differentiable in the origin, for starters. The two estimators use respectively the L1 and L2 norms, which are nonetheless related. However the estimator quality depends on the knowledge of the noise distribution, and different penalizations lead to different weighting of the noise. See the discussion in e.g. Ch.1 of Pattern Recognition and Machine Learning by C.Bishop

Hope this helps!