# Which multilevel software should I choose?

I have used packages in R thus far for MLM, but now I need to do MLM with complex survey data, and as I understand it, none of the MLM packages in R can cope with the complex weighting needed to analyze this type of data. (I have done regular regression/logistic regression with complex survey data in R using the survey package.)

So now I am looking for a different software package to purchase this month that would allow me to do this kind of analysis. I have been reading information online about the various programs, but I haven't been able to find, for example, videos of the kind of analysis that I am interested in in the different programs, that would give me an idea of how difficult it will be for me to figure out the different interfaces.

Here are the main considerations in my choice of software:

• I need to be able to do MLM with complex survey data, as explained above.
• I am also interested in doing multiple imputation and propensity score matching. I have done both of these things in R already, but I'd like to explore them using MLMs (with and without the complex survey data) as well. (I know that R can theoretically do multiple imputation with MLMs, but I have never personally been able to get it to work with MLMs in R.)
• I'm not a great programmer. I have a math background but no formal programming experience--everything I know about programming (or R for example), I have taught myself using books/the internet. I'm still not that great at using new packages in R, since I often have to educate myself about basic programming topics like data structures, etc. I also often have trouble linking how the programming commands or errors link at all to the underlying math. (I do like to be able to have some control over what I am doing, but that doesn't do me much good when I can't figure out how to program it correctly. :)) So the upshot is, I'm looking for something that will be straightforward (and hopefully not hugely time-consuming) to learn.
• The research I am currently doing is in the social sciences, in education, in case that helps.
• I also do a lot of logistic regression, in case that is relevant to choosing a package.

If you have any experience in this area and wouldn't mind sharing your advice on the matter, I'd be grateful!

• lmer in lme4 supports using weights. What is wrong with it for you? – Tim Dec 18 '14 at 10:19
• It doesn't support the kind of weighting used in complex surveys. Using the weights in lmer, as I understand it, will not properly adjust the standard errors as required for complex survey data (rather it assumes a simple random sample). – cww Dec 18 '14 at 15:56
• @cww is right; none of the R mixed model packages (written from a biostat perspective) do the weights social scientists need. So I don't think this is an opinion-based question: whether a package supports a given feature, such as proper complex survey weights, is a fact of life (reflecting opinions of the developer, if anything). – StasK Dec 18 '14 at 16:09
• I believe that the most recent version of SAS Proc Glimmix allows for survey weights. – Jeremy Miles Dec 18 '14 at 20:00
• Hi @JeremyMiles -- can you post a link to the documentation? – StasK Dec 19 '14 at 15:14

gllamm.org for Stata is the most flexible package to support multilevel weights. Per se, gllamm is only an estimation engine. You would need to provide your own steps in data transformation, including weight scaling. Of course, Stata has all of the multiple imputation, complex survey data analysis, matching. gllamm is written by a researcher from a school of education who published on using this package in complex survey data context, if that helps :). It will be time consuming to learn though, especially given that you need to learn a whole package whose paradigm is different from R.

• Thanks for the recommendation. I've been looking into this and considering this option based on your advice. I'm loathe to learn a whole new statistical program, but if it really is the best option for my needs, then it should pay off in the long run... – cww Dec 19 '14 at 11:21

I would recommend consulting the "Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis" which was published in 2014 for three relevant chapters and the references therein. Chapter 17 by Laura Stapleton is called "Incorporating Sampling Weights into Single- and Multilevel Analyses", Chapter 18, by Kim, Anderson, and Keller is called "Multilevel Analysis of Assessment Data", and Chapter 21, by Anderson, Kim, and Keller is called "Multilevel Modeling of Categorical Response Variables".

All three chapters give examples of how to incorporate design weights into multilevel modeling and address software capabilities. Chapters 21 and 18 also give example strategies for multiple imputation with multilevel data with sampling weights.

Since you do a lot of logistic regression (and will likely be doing more if you are interested in estimating propensity scores), chapter 21 will be of particular relevance. The key is to look for a program that uses adaptive quadrature to estimate model parameters; psuedolikelihood approaches tend to perform poorly with categorical outcomes and weights.

AFAIK, software options for multilevel logistic regression with design weights that use adaptive quadrature and report robust sandwich standard errors include Mplus, SAS NLMIXED, and gllamm (see StasK's answer).

• Answers can change order. References to "above" and "below" are highly fragile therefore. – Nick Cox Dec 18 '14 at 18:02
• Thanks for the reference. This is really helpful. I've been reading up on the three software programs that you mention (Mplus, SAS NLMIXED, gllamm), and trying to figure out which one would be best for my needs. (I do already have SAS, I just don't have much experience using it, since I have always used R. But I wouldn't want the fact that I already have it to be the main reason to go with that if another option is significantly better.) Just wondering if you have any insight about comparisons among these three, given my needs/preferences? – cww Dec 19 '14 at 11:21
• The Mplus syntax for incorporating design weights couldn't be simpler. If, as you say, you have trouble translating the underlying math to programming language, I would recommend Mplus for that reason. For example, to include level-one and level-two weights you simply add the commands WEIGHT = <instert variable name here>; and BWEIGHT = <insert variable name here>; to the VARIABLES section of the input and Mplus scales the weights as recommended by Pfeffermen et al 1998 JRSSB. – bsbk Dec 19 '14 at 17:19