GARCH and many other time series models are estimated using maximum likelihood methods (MLE). The common measure of in-sample fit is the information criterion such as Akaike (AIC) or Bayesian (BIC). These are computed using the loglikelihoods.
I sometimes use FVU, which may feel more familiar to you if you're used to $R^2$. Its problem is that it doesn't account for parsimony though, that's why it's my last resort tool when AIC is not applicable