Unbounded likelihoods for unpenalized mixed effects Most mixed effects models assume that the random effects $\gamma$ follow a $MVN(0, \Sigma_{\gamma})$ distribution. In some cases, specific structure is put on $\Sigma_{\gamma}$. For now, let's just assume it's an unstructured covariance matrix. 
Now, if we don't apply any priors or penalties on $\Sigma_{\gamma}$, it seems to me that the likelihood is unbounded. In particular, if we set $\gamma = 0$, then as $\det(\Sigma_{\gamma}) \rightarrow 0$, the contribution of the log-density of $\gamma$ approaches infinity. As long as the other contributions to the likelihood are finite (which is typically the case), this implies the log-likelihood would be unbounded. 
Clearly, for the MLE this creates an issue (although I know REML is a more popular alternative). For most MCMC algorithms, having unbounded log-densities can be problematic as well, even if the posterior is still proper. 
How is this issue typically handled? Is there a canonical penalty/prior on $\Sigma_{\gamma}$?
 A: Taking the simple model from amoeba
$$y_{ij} \sim N(\mu_i,\sigma_f^2) \qquad \text{with} \qquad \mu_i \sim N(0,\sigma_r^2)$$
The probability density to observe a sample of $\mathbf{ y_{ij} }$ is:
$$f_{\mathbf{ Y_{ij} }}(\mathbf{ y_{ij} }) =det((2\pi)^k\Sigma)^{-\frac{1}{2}}  e^{\mathbf{ y_{ij}^T\Sigma y_{ij}}}$$
With $\Sigma$ having a block structure like 
$$\Sigma = \begin{bmatrix}
J_1   & 0            &   \dots     &0       \\
0   & J_2   & \dots &  0        \\
\vdots & \vdots & \ddots & \vdots  \\
0  & 0      & \dots & J_n   \\
\end{bmatrix}$$
and the blocks are like
$$J_i = \begin{bmatrix}
\sigma_f^2+\sigma_r^2   & \sigma_r^2            &   \dots     & \sigma_r^2      & \sigma_r^2   \\
\sigma_r^2   & \sigma_f^2+\sigma_r^2   & \dots &  \sigma_r^2       & \sigma_r^2  \\
\vdots & \vdots & \ddots & \vdots &  \vdots \\
\sigma_r^2   &  \sigma_r^2       & \dots & \sigma_f^2+\sigma_r^2  & \sigma_r^2  \\
\sigma_r^2  & \sigma_r^2      &  \dots     & \sigma_r^2      & \sigma_f^2+\sigma_r^2 
\end{bmatrix} $$


Nothing goes wrong when $\sigma_r \to 0$ 
...except $f_{\mu_i}(0) \to \infty$ becomes a degenerate distribution, which is however not relevant for the calculation/expression of the distribution $f_{\mathbf{Y_{ij} }}$. 
If you consider the space of points $Y_{ij},\mu_i$ then you can see all the probability concentrating on a hyper-surface with $\mu_i=0$ and the density $f_{\mu_i}$ (along with $f_{Y_{ij},\mu_i}$) goes to infinity on this surface. But instead of $f_{Y_{ij},\mu_i}$ you wish to calculate $f_{Y_{ij}}$ $$f_{Y_{ij}}(y_{ij}) = \int f_{Y_{ij},\mu_i}(y_{ij},\mu_i) d\mu_i = \int f_{Y_{ij}|\mu_i}(y_{ij},\mu_i) f_{\mu_i}(\mu_i) d\mu_i$$ This density distribution $f_{Y_{ij}|\mu_i}$ for the distribution of $Y_{ij}$ on the hyper-surfaces with coordinates $\mu_i$ does not go to infinity. Or from another viewpoint, you integrate $f_{\mu_i}$ over an infinitely thin surface.

The case might be that you use the following probability density / likelihood:
$$f_\mathbf{y_{ij}}(\mathbf{y_{ij}}\vert \mathbf{\mu_i}, \sigma_f, \sigma_r) =  \frac{1}{\left( \sqrt{2 \pi \sigma_f^2} \right)^{n_j}} e^ { \frac{\sum_{j=1}^{n_{j}} (y_{ij}-\mu_i)^2}{2 \sigma_f^2} } \cdot \frac{1}{\left( \sqrt{2 \pi \sigma_r^2} \right)^{n_i}} e^{ \frac{\sum_{i=1}^{n_i} (\mu_i)^2}{2 \sigma_r^2} }   $$
but I would say that this is badly defined (the math looks ok, but the interpretation is not). This is not a density that only needs to be integrated over $d y_{ij}$, but also over $d \mathbf{\mu_i}$. You should not turn this into a likelihood function like $\mathcal{L}(\mathbf{\mu_i}, \sigma_f, \sigma_r \vert \mathbf{y_{ij}})$ but instead $\mathcal{L}( \sigma_f, \sigma_r \vert \mathbf{y_{ij}}, \mathbf{\mu_i})$ (yet you do not observe $\mathbf{\mu_i}$). 
It is incorrect to impose a relationship between the parameters in the likelihood function and add a corresponding density term to the expression of the likelihood function (that not surprisingly will blow up to infinity, in this way every unobserved variable may be added and becomes an infinite density somewhere, you could also add unobserved unicorns if you like).
