# Random effects Models vs Gaussian Log likelihood + explicit grouping features

Let us assume the Gaussian negative log likelihood (like e.g. here https://pytorch.org/docs/stable/generated/torch.nn.GaussianNLLLoss.html)

$$\text{Gaussian Negative Log Likelihood} = -\frac{1}{n} \sum_{i=1}^{n} \log\left(\frac{1}{\sqrt{2\pi\hat\sigma^2}} e^{-\frac{(x_i - \hat{\mu})^2}{2\hat\sigma^2}}\right)$$

to be minimized by a model, where the estimated mean $$\hat{\mu}$$ and the estimated variance $$\hat{\sigma}^2(x)$$ are computed by the model from the features $$x$$. Groups shall be encoded in the feature vector $$x$$ (e.g. by one-hot encoding).

Isn't this resulting in "kind of a mixed effect model" (even with instance wise variance if we pass more than the grouping features to the model head which computers the variance estimator)? This mixed effect modeling is indirect because there is no explicit random effects noise term $$Zx'$$ (like in $$y=Ax + Zx'+\epsilon$$) which is to be estimated, but instead this noise is adsorbed into a noise $$\epsilon'= \epsilon + Zx'$$ and the mean of $$Zx'$$ is adsorbed into the feature vector contribution $$Ax$$ (due to the one hot encodings of groups in $$x$$), so we have a model $$y =Ax + \epsilon'(x)$$ and $$\hat{\sigma}^2(x)$$ is the estimator of $$Var(\epsilon'(x)$$)

Am I right with this big picture or am I missing something? (I think that with sufficient care both kinds of models could be made equivalent. At least for Gaussian random effects and if we force $$x'$$ not to be parametrized by free parameters but to depend on x, I.e. $$x'(x)$$ ...., but I have not worked out ths details)

Please let me know if the question is clear or if I can clarify it more.

• Thank you Sextus Empiricus, your point is valid and I will have to post an update. In case of correlations I would use a Gaussian Negative log likelihood which is not assuming independent errors, but errors between observations with a correlation matrix $\Sigma$: $\ell(\vartheta=(\vec{\mu},\Sigma)) =\log f_{\vec{\mu},\Sigma}(\vec{x})=\log\left(\frac{1}{(2\pi)^{n/2}\sqrt{\det(\mathit\Sigma)}} \exp\left( -\frac{1}{2} \left[x_1-\mu_1,\ldots,x_n-\mu_n\right]\mathit\Sigma^{-1} \left[x_1-\mu_1,\ldots,x_n-\mu_n\right]^\mathrm{T} \right)\right),$ Commented Apr 27 at 7:42

In random effects models, if you absorb the random effects into the noise terms, then you get correlations between the noise terms. This is in contrast to the likelihood model which you describe where there are no correlations in the noise terms and all the terms are independent.

Correlations between the noises are not happening with your likelihood model if you only specify the model with variances $$\hat{\sigma}_i^2$$: this leaves the error terms independent from each other.

If you specify your likelihood model with the covariance matrix $$\Sigma$$ then you allow for correlation in the noises. If in addition you specify that covariance matrix with a specific group wise structure (see example below) then you have a random effects model.

Example for a random effects model:

$$y_{ij} \sim N(\mu_i,\sigma_f^2) \qquad \text{with random effect} \qquad \mu_i \sim N(0,\sigma_r^2)$$ The probability density to observe a sample of $$\mathbf{ y_{ij} }$$ is:

$$f_{\mathbf{ Y_{ij} }}(\mathbf{ y_{ij} }) =det((2\pi)^k\Sigma)^{-\frac{1}{2}} e^{\mathbf{ y_{ij}^T\Sigma y_{ij}}}$$

With $$\Sigma$$ having a block structure like

$$\Sigma = \begin{bmatrix} J_1 & 0 & \dots &0 \\ 0 & J_2 & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & J_n \\ \end{bmatrix}$$ and the blocks are like

$$J_i = \begin{bmatrix} \sigma_f^2+\sigma_r^2 & \sigma_r^2 & \dots & \sigma_r^2 & \sigma_r^2 \\ \sigma_r^2 & \sigma_f^2+\sigma_r^2 & \dots & \sigma_r^2 & \sigma_r^2 \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ \sigma_r^2 & \sigma_r^2 & \dots & \sigma_f^2+\sigma_r^2 & \sigma_r^2 \\ \sigma_r^2 & \sigma_r^2 & \dots & \sigma_r^2 & \sigma_f^2+\sigma_r^2 \end{bmatrix}$$

You can see random effects models as a special case of the likelihood model that you describe

and the estimated variance $$\hat{\sigma}^2(x)$$ are computed by the model from the features $$x$$

if it is applied to a covariance matrix instead of just the variance.

Here the random effects model is a special case, because it defines a particular structure for the dependency on the features, with the block structures like in the example above.

• +1 Thank you for your valuable insight and the confirmation that my intuition is right here. Surely, it makes sense to restrict the possible parametrizations of the covariance matrix $\Sigma$, otherwise we would have $1/2 n(n-1)>n$ to be estimated paramteres just for this matrix, not feasible LS. Could you expand a bit more why the likelihood formulation is less commonly known/used (based on what I find on the Internet) than "random effects models"? Is there deep drawbacks (I mean we can easily optimize the parameters under such a likelihood using stochastic gradient descent in pyTorch ...)? Commented Apr 27 at 8:02
• @Ggjj11 there is a formulation of random effects models with the exponential family distributions, called GLMM. It is, I believe, not the most usual form. Commented Apr 27 at 8:27
• @Ggjj11 the likelihood formulation is maybe less used because, as you say, the random effect term is only indirectly appearing in the likelihood formulation. The formula expression like y = Ax + Zx + e relates more to a mechanistic thinking about the random effects and shapes it like a multilevel model. Eventually, if you solve the equations, them you turn it into the likelihood formulation and the covariance matrix. Commented Apr 27 at 8:33
• @Ggjj11 here you see a manual optimization of a mixed effects model in R-code: stats.stackexchange.com/questions/337132 I can imagine that the software libraries that do this have some more optimal ways of obtaining the maximum likelihood, but the code at least gives an intuitive idea about what happens under the hood. (I remember the first time that I tried figuring out what mixed models do and manually fitted such model, then I wrongly used a multilevel approach and also fitted the random effects instead of integrating over them) Commented Apr 27 at 8:50
• This is the question that I was thinking of: „Can someone give an intuitive explanation how the (co)variance parameters of the random effects can be estimated without actually using/estimating the random effects?“ In Intuition about parameter estimation in mixed models (variance parameters vs. conditional modes) Commented Apr 28 at 7:31