What are the advantages and disadvantages of running separate models vs. multilevel modeling?
More particularly, suppose a study examined patients nested within doctors' practices nested within countries. What are the advantages/disadvantages of running separate models for each country vs. a three level nested model?
The question is dated but I think it's very important. The best answer I can get is from Joop J Hox (2010) book "Multilevel Analysis Techniques and Applications, Second Edition".
Suppose two-level hierarchical data with $p$ explanatory variables at the lowest level and $q$ explanatory variables at the highest level. Then, at page 55, he writes:
An ordinary single-level regression model for the same data would estimate only the intercept, one error variance, and p + q regression slopes. The superiority of the multilevel regression model is clear, if we consider that the data are clustered in groups. If we have 100 groups, estimating an ordinary multiple regression model in each group separately requires estimating 100 × (1 regression intercept + 1 residual variance + p regression slopes) plus possible interactions with the q group-level variables. Multilevel regression replaces estimating 100 intercepts by estimating an average intercept plus its residual variance across groups, assuming a normal distribution for these residuals. Thus, multilevel regression analysis replaces estimating 100 separate intercepts by estimating two parameters (the mean and variance of the intercepts), plus a normality assumption. The same simplification is used for the regression slopes. Instead of estimating 100 slopes for the explanatory variable pupil gender, we estimate the average slope along with its variance across groups, and assume that the distribution of the slopes is normal. Nevertheless, even with a modest number of explanatory variables, multilevel regression analysis implies a complicated model. Generally, we do not want to estimate the complete model, first because this is likely to get us into computational problems, but also because it is very difficult to interpret such a complex model. We prefer more limited models that include only those parameters that have proven their worth in previous research, or are of special interest for our theoretical problem.
That's for the description. Now the pages 29-30 will answer your question more accurately.
The predicted intercepts and slopes for the 100 classes are not identical to the values we would obtain if we carried out 100 separate ordinary regression analyses in each of the 100 classes, using standard ordinary least squares (OLS) techniques. If we were to compare the results from 100 separate OLS regression analyses to the values obtained from a multilevel regression analysis, we would find that the results from the separate analyses are more variable. This is because the multilevel estimates of the regression coefficients of the 100 classes are weighted. They are so-called Empirical Bayes (EB) or shrinkage estimates: a weighted average of the specific OLS estimate in each class and the overall regression coefficient, estimated for all similar classes.
As a result, the regression coefficients are shrunk back towards the mean coefficient for the whole data set. The shrinkage weight depends on the reliability of the estimated coefficient. Coefficients that are estimated with small accuracy shrink more than very accurately estimated coefficients. Accuracy of estimation depends on two factors: the group sample size, and the distance between the group-based estimate and the overall estimate. Estimates for small groups are less reliable, and shrink more than estimates for large groups. Other things being equal, estimates that are very far from the overall estimate are assumed less reliable, and they shrink more than estimates that are close to the overall average. The statistical method used is called empirical Bayes estimation. Because of this shrinkage effect, empirical Bayes estimators are biased. However, they are usually more precise, a property that is often more useful than being unbiased (see Kendall, 1959).
I hope it's satisfying.
Specifying a random effect involves assuming that the means of those levels are samples from a normal distribution. Better to specify them as fixed effects, AKA dummy variables if this assumption doesn't fit your data. In this way you are controlling for groupwise heterogeneity in the mean (at that level), but you are NOT allowing for heterogeneity in responses to your lower-level variables.
If you expect heterogeneity in response to your lower-level explanatory variables, separate models make sense, unless you want to run some sort of random coefficient model (which again involves the assumption that coefficients are normally distributed).
(I believe there are methods for non-normal random effects, but nothing as widely used or accessible as lme)
Advantage: The ability to explicitly test for differences in parameters by cluster (i.e. differences in significance do not mean significant differences).
