Random effects Models vs Gaussian Log likelihood + explicit grouping features

Question

Let us assume the Gaussian negative log likelihood (like e.g. here https://pytorch.org/docs/stable/generated/torch.nn.GaussianNLLLoss.html)

$\text{Gaussian Negative Log Likelihood} = -\frac{1}{n} \sum_{i=1}^{n} \log\left(\frac{1}{\sqrt{2\pi\hat\sigma^2}} e^{-\frac{(x_i - \hat{\mu})^2}{2\hat\sigma^2}}\right)$

to be minimized by a model, where the estimated mean $\hat{\mu}$ and the estimated variance $\hat{\sigma}^2(x)$ are computed by the model from the features $x$. Groups shall be encoded in the feature vector $x$ (e.g. by one-hot encoding).

Isn't this resulting in "kind of a mixed effect model" (even with instance wise variance if we pass more than the grouping features to the model head which computers the variance estimator)? This mixed effect modeling is indirect because there is no explicit random effects noise term $Zx'$ (like in $y=Ax + Zx'+\epsilon$) which is to be estimated, but instead this noise is adsorbed into a noise $\epsilon'= \epsilon + Zx'$ and the mean of $Zx'$ is adsorbed into the feature vector contribution $Ax$ (due to the one hot encodings of groups in $x$), so we have a model $y =Ax + \epsilon'(x)$ and $\hat{\sigma}^2(x)$ is the estimator of $Var(\epsilon'(x)$)

Am I right with this big picture or am I missing something? (I think that with sufficient care both kinds of models could be made equivalent. At least for Gaussian random effects and if we force $x'$ not to be parametrized by free parameters but to depend on x, I.e. $x'(x)$ ...., but I have not worked out ths details)

Please let me know if the question is clear or if I can clarify it more.

Answer 1

In random effects models, if you absorb the random effects into the noise terms, then you get correlations between the noise terms. This is in contrast to the likelihood model which you describe where there are no correlations in the noise terms and all the terms are independent.

Correlations between the noises are not happening with your likelihood model if you only specify the model with variances $\hat{\sigma}_i^2$: this leaves the error terms independent from each other.

If you specify your likelihood model with the covariance matrix $\Sigma$ then you allow for correlation in the noises. If in addition you specify that covariance matrix with a specific group wise structure (see example below) then you have a random effects model.

Example for a random effects model:

$$y_{ij} \sim N(\mu_i,\sigma_f^2) \qquad \text{with random effect} \qquad \mu_i \sim N(0,\sigma_r^2)$$ The probability density to observe a sample of $\mathbf{ y_{ij} }$ is:

$$f_{\mathbf{ Y_{ij} }}(\mathbf{ y_{ij} }) =det((2\pi)^k\Sigma)^{-\frac{1}{2}} e^{\mathbf{ y_{ij}^T\Sigma y_{ij}}}$$

With $\Sigma$ having a block structure like

$$\Sigma = \begin{bmatrix} J_1 & 0 & \dots &0 \\ 0 & J_2 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & J_n \\ \end{bmatrix}$$ and the blocks are like

$$J_i = \begin{bmatrix} \sigma_f^2+\sigma_r^2 & \sigma_r^2 & \dots & \sigma_r^2 & \sigma_r^2 \\ \sigma_r^2 & \sigma_f^2+\sigma_r^2 & \dots & \sigma_r^2 & \sigma_r^2 \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ \sigma_r^2 & \sigma_r^2 & \dots & \sigma_f^2+\sigma_r^2 & \sigma_r^2 \\ \sigma_r^2 & \sigma_r^2 & \dots & \sigma_r^2 & \sigma_f^2+\sigma_r^2 \end{bmatrix} $$

---

You can see random effects models as a special case of the likelihood model that you describe

and the estimated variance $\hat{\sigma}^2(x)$ are computed by the model from the features $x$

if it is applied to a covariance matrix instead of just the variance.

Here the random effects model is a special case, because it defines a particular structure for the dependency on the features, with the block structures like in the example above.