Linear mixed models nested random effects: can you nest within a factor that has just 2 observations? I'm working in the context of a twin study where I have to account for the relatedness between twins in the same family. The way I usually do this is by specifying a linear mixed model with lmer similar to this: lmer(x ~ y + (1|familyID), dat)
My understanding is that because there's only 2 participants in each family, I can't specify random slopes because there are insufficient degrees of freedom. Now I'm wondering if this has any impact on whether you can nest other random effects within the family random effect. For instance, if I ran a between by within experiment in the twin sample, would I be able to nest the within-subjects component within each family, or would the lack of degrees of freedom within the family group cause this model to be over-specified? Or would this not be an issue as long as there were multiple measurements per subject?
Thanks
 A: There seems to be some confusion.

My understanding is that because there's only 2 participants in each family, I can't specify random slopes because there are insufficient degrees of freedom

This does not make sense to me. In general, random slopes do not make sense when the variable in question does not vary with subjects. So if you have repeated measures within levels of a grouping variable, then you can, in principle, fit random slopes (provided that they are supported by the data)
On the other hand, depending on what level you are taking your measurements / observations, the model may be mispecified. If you are measuring variables at the family level - eg father's ethnicity or mother's education level; or if you are making repeated measures at the family level - eg annual household income over several years, or family address (which may change over time), then the proposed model should be a good place to start. However if you are making repeated measures per twin, then you will need to fit random intercepts for twin ID, varying within family:
lmer(x ~ y + (1|familyID/twinID), dat)

Regarding the issue of nesting:

I'm wondering if this has any impact on whether you can nest other random effects within the family random effect.

There is no reason why you can't have further random effects nested within family. For example, as mentioned above, if you have repeated measures within individual twins then you would fit nested random effects. Although there are only 2 twins in each family, when fitting nested random effects it is the number of levels of the upper level factor that is important, since:
x ~ y + (1|familyID/twinID)

is exactly the same ae
x ~ y + (1|familyID) + (1 |familyID:twinID)

There will always be more levels of familyID:twinID than just familyID, so the constraint in terms of group sizes is familyID, not twinID
