As stated here, we usually do not need to estimate the variance-covariance matrix of the error term (or residuals, in the sense of latent outcome) in generalized linear mixed models:
The variance-covariance matrix of the residuals, $ε$ or the condition
covariance matrix of $y|X\beta+Z\gamma$. The most common residual
covariance structure is
$$R=I\sigma^2_ε$$ where $I$ is the identity matrix (diagonal matrix of
1s) and $\sigma^2_ε$ is the residual variance. This structure assumes
a homogeneous residual variance for all (conditional) observations and
that they are (conditionally) independent. Other structures can be
assumed such as compound symmetry or autoregressive.
Take binary data for an example, for logit link function, $\sigma^2_ε=\pi^2/3$ as the error comes for standard logistic distribution; for probit link, $\sigma^2_ε=1$ as the error comes from standard normal distribution.