Speaking about evaluating volatility forecasts in general (not GARCH in specific), I will mention an alternative to Stephan Kolassa's answer.
One can also study proper scoring rules for statistics or "properties" of distributions; this area is sometimes called elicitation. There, one can ask the following question: Is there a "proper" scoring rule $S(v, y)$ that evaluates a forecast $v$ of the variance of a random variable using a sample $y$? Here the notion of proper should be that expected score is maximized when $v$ is the true variance.
It turns out that the answer is no. However, there is a trick. There is certainly such a scoring rule for the mean, e.g. $S(u, y) = - (u - y)^2$. It follows that there is a scoring rule for the second moment (not centered), e.g. $S(w, y) = - (w - y^2)^2$.
Therefore, to evaluate a forecast of variance in an unbiased way, it suffices in this case to query the forecast for just two parameters, the first and second moments, which determine the variance. In other words, it's not actually necessary to produce and evaluate the full distribution. (This is basically your proposal: we first evaluate the conditional mean, then the residual, roughly.)
There are of course other measures of volatility than variance, and there is research on whether they are "directly elicitable" (i.e. there exists a proper scoring rule eliciting them) or, if not, their "elicitation complexity" (i.e. how many parameters must be extracted from the underlying distribution in order to evaluate it). One place this is studied is for risk measures in finance. The statistics studied include expectiles, value-at-risk, and conditional-value-at-risk.
There is some general discussion in Gneiting, Making and Evaluating Point Forecasts, Journal of the American Statistical Association (2011). https://arxiv.org/abs/0912.0902 . Elicitation complexity is studied in Frongillo and Kash, Vector Valued Property Elicitation, Conference on Learning Theory (COLT, 2015). http://proceedings.mlr.press/v40/Frongillo15.html