I can't be the first person to think about estimating variance components of mixed models by gradient descent, and then computing BLUP's at each update. Googling, I find little on the topic. But the gradients seem tractable (if messy). I suppose one would be logging the variances to render them continuous. I also suppose one would need to penalize the loss function, otherwise the variances would go to infinity.

Can anyone provide any further insight before I go and program it? References? Software? Tips/tricks for optimization? Would minibatch or stochastic gradient descent be problematic?


I would first look into the optimization methods available in R' lme4: See for example Section 4.2 of [1] and the lme4 paper [2].

The penalized least squares algorithm seems to be using gradient information to provide an iterative solution.

[1] https://cran.r-project.org/web/packages/lme4/vignettes/Theory.pdf

[2] https://www.jstatsoft.org/article/view/v067i01/v67i01.pdf


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.