GARCH vs ARCH models - which is more parsimonious? As I understand the inclusion of the GARCH term, $\sigma^2$, in a GARCH model allows for an infinite number of time series terms, $\epsilon^2$, to influence the conditional variance. Is this the case? How does this characteristic enable the GARCH model to be more parsimonious than the ARCH model?  
 A: 
As I understand the inclusion of the GARCH term, $\sigma^2$, in a GARCH model allows for an infinite number of time series terms, $\epsilon^2$, to influence the conditional variance. Is this the case?

Yes, it is. A GARCH model can be expressed (under some regularity condition) as an infinite-order ARCH model, thus making the conditional variance $\sigma_t^2$ depend on an infinite number of past squared innovations $\varepsilon_{t-i}^2$ for $i=1,2,\dots$. 

How does this characteristic enable the GARCH model to be more parsimonious than the ARCH model?

Citing the original paper introducing the GARCH model (Bollerslev, 1986),

Common to most of the [ARCH model] applications however, is the introduction of a rather arbitrary linear declining lag structure in the conditional variance equation to take account of the long memory typically found in empirical work, since estimating a totally free lag distribution often will lead to violation of the non-negativity constraints. <...> In this light it seems of immediate practical interest to extend the ARCH class of models to allow for both a longer memory and a more flexible lag structure. 

(emphasis is mine).
Under long memory, the influence of past innovations dies out gradually and slowly, and any finite-order ARCH model fails to capture this efficiently. On the other hand, the GARCH model excels at this. How? See the answer to your first questions above. Thus GARCH is more parsimonious as it uses just a couple of (or a few) parameters to achieve what the ARCH model would need an infinite number of parameters for. The argument is also very similar (essentially the same) to how an ARMA model is more parsimonious than an AR or an MA model.
References:


*

*Bollerslev, T. (1986). Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics, 31(3), 307-327.

