How is the standard deviation of random effects estimated? For example, the sd of the random intercept reported by lme4 when I use lmer or glmer is much higher than if I just calculated the sd on the list of intercepts generated from ranef. Why is this?
 A: Suppose that your model is something like
$$
y_{ij} = \beta + u_j + \varepsilon_{ij}
$$
where $u_j \sim \mathcal{N}(0, \sigma^2_u)$, for $j=1,\ldots,m$, are the random intercepts and $\varepsilon_{ij} \sim \mathcal{N}(0, \sigma^2_\varepsilon)$ are the error terms. To keep this simple, let's assume that $i=1,\ldots,n$, i.e. there are $n$ observations in each of the $m$ groups/clusters.
Estimating the within-cluster variance ($\sigma^2_\varepsilon$) from the residuals is easy; we just take
$$
\hat{\sigma}^2_\varepsilon=\frac{1}{m(n-1)}\sum_{i,j}(y_{ij} - \bar{y}_{.j})^2
$$
where $\bar{y}_{.j}$ is the sample mean of the $j$th cluster. The denominator $m(n-1)$ comes from the fact that we lose one degree of freedom in each cluster.
To estimate the variance of the random intercepts ($\sigma^2_u$), we need to work a bit harder. The maximum-likelihood estimator is
$$
\hat{\sigma}^2_u=\frac{1}{m}\sum_{j}(\bar{y}_{.j} - \bar{y}_{..})^2 - \frac{1}{n}\hat{\sigma}^2_\varepsilon
$$
where $\bar{y}_{..}$ is the (global) sample mean. Why the two terms? The first one looks at the squared difference between the mean of the $j$th cluster and the overall mean, while the second one is an adjustment for the within-cluster variance.
