Random effects with random slopes and intercepts LMM Let's say we have two models specified by the following formulas in R's lmer():
i)   Y ~ A + B + (A:B|SUBJECT)
ii)  Y ~ A + B + (SUBJECT|A:B)
For the random effects, equation i) specifies a random slope and intercept for each level of A:B by subject, as equation ii) specifies a random slope and intercept for each subject by level of A:B (if i am not interpreting this wrongly).
For both these model formulas, what are the differences in the linear mixed effects model equation regarding the random effects? And is the model ii) wrongly specified? 
 A: In model i) the right side of | specifies that observations are nested/grouped within SUBJECT, resulting in the estimation of a random intercept (that is, for each level of SUBJECT, that level's intercept's deviation from the fixed intercept). The left side of the | specifies  that the interaction between A and B varies for each level of SUBJECT, that is, a random slope for the interaction. Normally this would be the deviation from the fixed estimate of the interaction. However the fixed part of your model formula does not contain the interaction, so in this case it is the deviation from zero.
In model ii) you have the interaction between A and B as the grouping variable, resulting in a random intercept for the interaction, and a random slope for SUBJECT so that the effect of SUBJECT varies for each level of the interaction.
Both models will also estimate a correlation between the random slope deviations and random intercept deviations.
Note that model i) is a little strange because you don't have the interaction as a fixed effect. A more common situation would be to have one or both both main effects as fixed effects and random effects, and the interaction as a fixed effect, both, or not at all.
Model ii) is rather more strange because it seems unlikely that the interaction could be considered as a grouping variable.
