Mixed-modeling when no observation contributes both X and Y

I'm working on a project investigating the relationship between (let's say) a face's perceived masculinity and its perceived competence. There was a large number of face stimuli (80). Two completely separate groups of subjects rated the faces on each dimension: a given subject rated a random subset of the 80 faces on EITHER masculinity OR competence, never both. In other words, every subject rated multiple faces on a single dimension, and every face was rated by multiple subjects on both dimensions. (This was done to avoid interference effects and demand bias.) The specific face stimulus is considered a nuisance parameter here; face-specific effects are not really of interest.

A very simple modeling approach would be to treat face as the unit of analysis and characterize each face by its mean masculinity and its mean competence. This would result in a dataset with 1 row per face that would be easily amenable to linear regression, etc.

Obviously, this approach loses a lot of information, particularly failing to account for a subject random effect (such that, e.g., some subjects might generally consider faces more competent than other subjects). The problem invites a mixed-model with random intercepts or slopes by subject and face plus a fixed effect of masculinity, such as (in R shorthand):

competence ~ masculinity + (1|id) + (1|face)

However, I am not sure how this would work in light of the experimental design. I am basically having trouble envisioning how the fixed effect of masculinity will be estimable given that no observation contributes both a masculinity AND a competence rating.

Is this problem conducive to mixed-modeling? If so, how?

UPDATE

I tried fitting the mixed model specified as above using lmer to see what would happen. Not surprisingly, the model throws an error about incorrect grouping structure. This error goes away if I randomly simulate some of the missing-by-design such that there are now observations having both X and Y.

So, I am revising my question:

Is there a modeling approach for this situation that would retain more information than just modeling the face-specific means?

Because subjects (id) do not have a measure of both masculinity and competence, you cannot, as you mention, associate them at the subject level. That is, you cannot test whether people who perceive a face as masculine also perceive it as more or less competent. Instead, you have to examine the relationship of these variables at the level of the face because each measure was taken for the different faces.
If you want to use mixed models, it seems that you should specify the model with a random intercept for id partially crossed with face. I am not so familiar with lmer, but it looks like you are specifying this correctly. This documentation would help with specification. Under this specification, you are coming to conclusions more along the lines of "faces that are perceived to be [more or less] masculine tend also to be perceived as [more or less] competent."