How can I test for differences in variation between groups in a mixed model (lme4)? I would like to test for differences in variation, not in means, between two sites. By looking at a boxplot of my data I see that bird song in one site look much more variable in length than in another site, and also it seems there is more variation within individuals in one site than in the other. In other words, more variation within and between individuals (or so it looks..).
I'm fitting this random model to test for differences:
a   <- lmer (song_length~factor(zone)+date+(1|male/songtype),
            data, REML=TRUE)

The reason for the songtype random factor is that birds have different types of songs, and I take measurements of 4 songs per song type per bird.
The question would be: is there more inter- and/or intraindividual variation in site A than in site B?
Also, if there is indeed heteroscedasticity, the model is wrong, and I would need to account for that. But that is a second question I guess.
 A: A mixed model may not be appropriate for your research question (about the difference in variance between groups).
Mixed models are used for inference and prediction in data where random effects are assumed to be present, such as in hierarchical data. Typically the interest is in modelling fixed effects while controlling for random effects or in the random effect themselves (such in understanding the variability across levels through variance components). 
In most software, such as lme4, it is assumed that there is homoscedasticity. In some software heteroscedasticity can be modeled using weights. But there is no way to test for a difference in variation between groups in the model. Your question is about variation between groups, so one approach is simply to perform an F test of equal variance between site A and site B. 
Your model formulation suggests that you are interested in the association between song_length and zone (fixed effect) while controlling for the confounder (date) and the random effects (male and songtype). I don't immediately see how gender can be a random effect - I would expect this to be a fixed effect.
A: A few points:


*

*sometimes log-transforming data can clear up heteroscedasticity nicely; this would be my first attempt, especially as you have a positive responses variable (song length), so we would expect it to be skewed if the coefficient of variation is large

*assuming that zone in your model is equivalent to "site" in your verbal description (i.e. "site A vs. site B", is coded in the zone variable in your data), you can use the weights argument in nlme::lme to add heteroscedasticity (within-individual variability) to the model:
a <- lme(song_length~factor(zone)+date,
     random=~1|male/songtype,
     data, method="REML",
     weights=varIdent(~zone))

You can do this a little bit more awkwardly with lmer by adding an observation-level variable and using a dummy variable to set its value to zero in the first site:
data <- transform(data,
          obs=factor(1:nrow(data)),
          zonedummy=as.numeric(zone=="B"))
a <- lmer(song_length~factor(zone)+date+
     (1|male/songtype) + (zonedummy-1|obs), 
     data, REML=TRUE,
     control=lmerControl(check.nobs.vs.nRE  = "ignore",
                         check.nobs.vs.nlev = "ignore"))

You can do a similar thing for the among-bird variation by adding a term of the type (zonedummy-1|male); this adds an additional among-male variance term only in site B (you may have to use additional arguments to lmerControl to override some errors).
