A very good article explaining the general approach of LMMs and their advantage over ANOVA is:
Linear mixed-effects models (LMMs) generalize regression models to have residual-like components, random effects, at the level of, e.g., people or items and not only at the level of individual observations. The models are very flexible, for instance allowing the modeling of varying slopes and intercepts.
LMMs work by using a likelihood function of some kind, the probability of your data given some parameter, and a method for maximizing this (Maximum Likelihood Estimation; MLE) by fiddling around with the parameters. MLE is a very general technique allowing lots of different models, e.g., those for binary and count data, to be fitted to data, and is explained in a number of places, e.g.,
- Agresti, A. (2007). An Introduction to Categorical Data Analysis (2nd Edition). John Wiley & Sons.
LMMs, however, can't deal with non-Gaussian data like binary data or counts; for that you need Generalized Linear Mixed-effects Models (GLMMs). One way to understand these is first to look into GLMs; also see Agresti (2007).