For most datasets I've worked with I've been fortunate enough to be able to handle the data in memory. However what if that data set is very large? Are there any model's or should I say "popular" models that typically perform well but are untrainable or can't be trained with "batches" and require all data at once?
1 Answer
$\begingroup$
$\endgroup$
Random effects models require having all of the data in memory for most estimation methods apart from ANOVA-based estimators.
Historically, ANOVA-based estimators were used when calculations were done by hand or when computing power and memory were more expensive and harder to come by. Unfortunately, they often can yield negative estimates of variances -- which is why we now prefer other estimation methods.
