30
$\begingroup$

I was wondering if anyone could enlighten me on the current differences between these two functions. I found the following question: How to choose nlme or lme4 R library for mixed effects models?, but that dates from a couple of years ago. That's a lifetime in software circles.

My specific questions are:

  • Are there (still) any correlation structures in lme that lmer doesn't handle?
  • Is it possible/recommended to use lmer for panel data?

Apologies if these are somewhat basic.

A bit more detail: panel data is where we have multiple measurements on the same individuals, at different points in time. I generally work in a business context, where you might have data for repeat/long-term customers over a number of years. We want to allow for variation over time, but clearly fitting a dummy variable for each month or year is inefficient. However, I'm unclear whether lmer is the appropriate tool for this sort of data, or whether I need the autocorrelation structures that lme has.

$\endgroup$
  • 1
    $\begingroup$ That answer is still up to date. lmer still does not handle the variety of correlation and variance structures that lme does, and as I understand the situation, it probably never will. $\endgroup$ – Aaron - Reinstate Monica Jul 13 '13 at 15:14
  • $\begingroup$ @Aaron Thanks for the reply. For the second part, does this affect lmer's ability to handle panel dataset? Or can I get away without making specific correlation assumptions? $\endgroup$ – Hong Ooi Jul 13 '13 at 15:16
  • 3
    $\begingroup$ @Aaron, I don't know about "never will" handle corr/var structures -- I am interested in adding these features and don't think it would be that difficult -- but I would certainly say "don't hold your breath". I'm not familiar enough with panel data to know what would be required for lmer to handle them ... Hong, can you add a brief explanation to the question that describes the necessary statistical properties in a little more detail, or gives pointers? $\endgroup$ – Ben Bolker Jul 13 '13 at 15:31
  • $\begingroup$ @BenBolker Added some details. $\endgroup$ – Hong Ooi Jul 13 '13 at 15:37
  • 4
    $\begingroup$ I would say lmer would be pretty good with a random effect of year and a random effect of customer (let's say you only have one measurement per customer per year); if you are fitted an overall (fixed-effect) trend of time you should also consider a random time-by-customer interaction (i.e. random slopes). Ideally you would also want to allow for temporal autocorrelation within each customer's time series, which is at the moment not possible with lmer, but you could check the temporal autocorrelation function to see if that was important ... $\endgroup$ – Ben Bolker Jul 13 '13 at 15:50
15
+100
$\begingroup$

UPDATE JUNE 2016:

Please see Ben's blog entry describing his current thoughts on accomplishing this in lme4: Braindump 01 June 2016

If you prefer Bayesian methods, the brms package's brm supports some correlation structures: CRAN brms page. (Note especially: "As of brms version 0.6.0, the AR structure refers to autoregressive effects of residuals to match the naming and implementation in other packages such as nlme. Previously, the AR term in brms referred to autoregressive effects of the response. The latter are now named ARR effects and can be modeled using argument r in the cor_arma and cor_arr functions.")


ORIGINAL ANSWER JULY 2013:

(Converted from a comment.)

I would say lmer would be pretty good with a random effect of year and a random effect of customer (let's say you only have one measurement per customer per year);

lmer(y~1 + (1|year) + (1|customer), ...)

would fit the (intercept-only) model

$$ Y_{ij} \sim \text{Normal}(a + \epsilon_{\text{year},i} + \epsilon_{\text{customer},j}, \sigma^2_0) $$ where $\epsilon_{\text{year}}$ and $\epsilon_{\text{customer}}$ are zero-mean Normal variates with their own specific variances.

This is a pretty boring model, you might want to add an overall (fixed-effect) trend of time and also consider a random time-by-customer interaction (i.e. random slopes). I think

lmer(y~year + (1|year) + (year|customer), ...)

should fit the model $$ Y_{ij} \sim \text{Normal}((a + \epsilon_{\text{customer},j}) + (b + \epsilon_{\text{year} \times \text{customer},j}) \cdot \text{year} + \epsilon_{\text{year},i}, \sigma^2_0) $$

(using year in this way is an exception to the usual rule of not including an input variable as both a fitted and a random effect in the same model; provided it's a numeric variable, year gets treated as continuous in the fixed effect and the year:customer (random) interaction and as categorical in the random effect ...)

Of course you might want to add year-level, customer-level, and observation-level covariates which would soak up some of the relevant variance (e.g. add average consumer price index to explain why years were bad or good ...)

Ideally you would also want to allow for temporal autocorrelation within each customer's time series, which is at the moment not possible with lmer, but you could check the temporal autocorrelation function to see if that was important ...

Caveat: I don't know that much about standard approaches for handling panel data; this is based just on my knowledge of mixed models. Commenters (or editors) should feel free to chime in if this seems to violate standard/best practices in econometrics.

$\endgroup$
  • $\begingroup$ Unless this is odd notation - usually $X \sim N(\mu, \sigma^2)$ means the marginal distribution of $X$ is normal with mean $\mu$ and variance $\sigma^2$ - I think your equations are not quite right. What you've written are conditional distributions, given the random effects. The marginal distribution of $Y_{ij}$ in the first model is $$N(a,\sigma^{2}_0 + \sigma^{2}_{year} + \sigma^{2}_{cust})$$ In the second model the marginal mean is $a+b \cdot {\rm year}$ and the variance is a more complicated expression involving the covariance between the year random slope/intercept plus the other stuff. $\endgroup$ – Macro Jul 13 '13 at 18:05
  • $\begingroup$ Yep, thanks Ben. In practice there would be fixed effects as well, eg age, sex and all the usual suspects. @Macro: Ben has it right, I believe. $\endgroup$ – Hong Ooi Jul 13 '13 at 18:24
  • $\begingroup$ @Macro: I think the notation is odd/unusual, but correct (i.e. equivalent to what you suggest.) I've expressed the random effect terms as part of $\mu$. It would probably be clearer/more familiar if I wrote it out in multilevel notation ($Y \sim \text{Normal}(X \beta + Z u, \sigma^2); u \sim \text{MVNormal}(0,\Sigma); \Sigma=f(\theta)$). $\endgroup$ – Ben Bolker Jul 13 '13 at 18:30
  • 1
    $\begingroup$ @BenBolker: have you noticed that I have set a bounty on this question, because I'm looking for a potential update? $\endgroup$ – S. Kolassa - Reinstate Monica May 25 '16 at 16:27
  • 2
    $\begingroup$ I've just posted some stuff I've been working on recently at rawgit.com/bbolker/mixedmodels-misc/master/notes/… ; I'll try to get around to incorporating relevant bits in my answer (alternatively, anyone else is welcome to either post their own answer based on that information, or edit my qestion!) $\endgroup$ – Ben Bolker May 25 '16 at 22:32
3
$\begingroup$

To answer your questions directly, and NB this is years after the original post!

  • Yep there are still correlation structures that nlme handles which lme4 will not handle. However, for as long as nlme allows the user to define general corstrs and lme4 does not, this will be the case. This has surprisingly little practical impact. The "big three" correlation structures of: Independent, Exchangeable, and AR-1 correlation structures are easy handled by both packages.

  • It's certainly possible. You can fit panel data with the lm function too! My recommendation about which to use depends on the problem. lme4 is a much smaller tool kit, and the formula representation is a neat, concise way of depicting some very common mixed effects models. nlme is the very large tool box, including a TIG welder to make any tools you need.

You say you want to allow for "variation over time". Essentially, an exchangeable correlation structure achieves this, allowing for a random intercept in each cluster, so that the intracluster variance is the sum of cluster level variation as well as (what you call) variation over time. And this by no means deters you from using fixed effects to obtain more precise predictions over time.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.