How to specify factors nested within treatments using a glmm I have behavioral response data (counts of acts within an hour trial) that I would like to analyze using a GLMM. I'm new to GLMM's but after months of digging through papers and forums, I've concluded this is likely the best approach to analyze my data.
The experiment was an exposure study where small estuarine crabs were exposed to fluoxetine over a 60-day study. We observed their behaviors using ethograms during day and night trials. We also observed their behaviors day and night with a predator added to the tank. There were 4 trials per week (Day -Pred, Day +Pred, Night -Pred, Night +Pred). Our question was whether exposure to the drug ultimately altered their behaviors (being still, mobile, foraging, predator avoidance, etc.) during these trials and if there was an effect of exposure time (i.e., greater effect with longer exposure)?
Because the data are counts within a specified trial period there are observed proportions (i.e., observed behavioral acts / 12 possible observation windows) I believe I should use either a Poisson or a binomial distribution to fit the model. We decided to subset the main dataset into trial types because day/night and +Pred/-Pred aren't treatments per se, so we constructed models for each subset rather than using one over-arching model. 
For example we built a model to compare the active behaviors across treatments within the Day+Predator subset: 
For reference the variable terms are: 


*

*actsuc= successful active behaviors

*actfail= active behavior failures


Fixed effects: 


*

*Treatment = 3 levels: Control, 3ng, 30ng fluoxetine; 

*exposure = 20 days, 40 days, 60 days; 

*crabsex = Dominant male, subordinate female, subordinate male;


Random effects: 


*

*TankID= to account for non-independence of crab within the same tank (3 in each)

*Trial = to account for non-independence of multiple trials over exposure study

*Crab. = to account for non-independence of multiple observations on a single crab within and across trials 


Model with interaction terms 
actfit.glmm.interact = glmer(cbind(actsuc, actfail) ~ Treatment:exposure +  
                             Treatment:crabsex + (1|TankID) + (1|Trial) + (1|Crab.), 
                             family=binomial, data=pday)

I need help understanding interaction and nested notation using glmer. I'm not sure if I've set this up correctly. I get the following error message:
fixed-effect model matrix is rank deficient so dropping 1 column / coefficient
Warning message:
In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv,  :
  Model failed to converge with max|grad| = 0.0247253 (tol = 0.001, component 6)

Further, I cannot use the predict function with this model as it is. See script and resultant error message below:
pday$predict.act <- predict(actfit.glmm.interact, newdata = pday,  re.form=NA, 
                            type="response")

Error in X %*% fixef(object) : non-conformable arguments

I have used the 'full' version of this model without interactions among fixed factors and was able to get predicted values without any error message:
actfit.glmm.full = glmer(cbind(actsuc, actfail) ~ Treatment + exposure + crabsex + 
                         (1|TankID) + (1|Trial) + (1|Crab.), family=binomial, data=pday)

Is this more appropriate, to treat the fixed effects without an interaction? I would like to compare behaviors across genders (crabsex) between treatments, but would like to know if the effect changes with exposure period (which I would assume would require an interaction). 
I've seen in previous posts that its useful to upload data or a screen shot of the dataframe for reference. Once I figure out how to do that I will upload here. 
 A: Does each crab appear only in one tank ?
Does each tank appear only in one trial ?
If the answer to both of these is yes then you have a nested design for which you can specify the random structure as (1|Trial/TankID/Crab), assuming you have sufficient number of trials, tanks and crabs (it wasn't clear from your question how many of each you have) - see the answer here for more details.
As for the interactions, if you think that the effect of Treatment is different for different levels of exposure then you would include the interaction term Treatment:exposure. Similarly for the other interaction. 
There is a warning:

fixed-effect model matrix is rank deficient so dropping 1 column / coefficient

There is probably linear dependence among your fixed effects. You can investigate this with:
require(caret)
findLinearCombos(model.matrix(actfit.glmm.interact))

The warning about convergence should also be checked, but you should write another question about that (after searching for possible answers, since this is a common problem)
