There seems to be some confusion.
My understanding is that because there's only 2 participants in each family, I can't specify random slopes because there are insufficient degrees of freedom
This does not make sense to me. In general, random slopes do not make sense when the variable in question does not vary with subjects. So if you have repeated measures within levels of a grouping variable, then you can, in principle, fit random slopes (provided that they are supported by the data)
On the other hand, depending on what level you are taking your measurements / observations, the model may be mispecified. If you are measuring variables at the family level - eg father's ethnicity or mother's education level; or if you are making repeated measures at the family level - eg annual household income over several years, or family address (which may change over time), then the proposed model should be a good place to start. However if you are making repeated measures per twin, then you will need to fit random intercepts for twin ID, varying within family:
lmer(x ~ y + (1|familyID/twinID), dat)
Regarding the issue of nesting:
I'm wondering if this has any impact on whether you can nest other random effects within the family random effect.
There is no reason why you can't have further random effects nested within family. For example, as mentioned above, if you have repeated measures within individual twins then you would fit nested random effects. Although there are only 2 twins in each family, when fitting nested random effects it is the number of levels of the upper level factor that is important, since:
x ~ y + (1|familyID/twinID)
is exactly the same ae
x ~ y + (1|familyID) + (1 |familyID:twinID)
There will always be more levels of familyID:twinID
than just familyID
, so the constraint in terms of group sizes is familyID
, not twinID