My question is about generalizability theory. A study by Bergeron, Floyd, McCormack, and Farmer (2008) is only adding to this confusion. Basically, in G-theory you want the first model you run to include all main effects and valid interactions. This is fairly straightforward for a crossed design, but nesting makes this more difficult for me to understand. I was reading Bergeron et al., 2008, and this is adding to the confusion.
They stated, on page 97,
We used a three-facet, partially nested design, with students and raters nested within classrooms and classrooms crossed with occasions and instruments. Variance components were computed for five main sources of variation (classroom, student-within-classroom, rater-within-classroom, occasion, and instrument), four two-way interactions (student-by-rater-within-classroom, classroom-by-occasion, classroom-by-instrument, and occasion-by-instrument), and residual (unexplained) error.
First, it sounds like they have five facets (factors), not three: student, rater, classroom, occasion, and instruments. Is that correct? or because the student and rater are nested, then you cannot consider them a true facet?
Second, the variance components in their model aren't all valid possible combinations, I think they are missing some?
- Bergeron main effects as reported:
- Bergeron second level effects as reported:
- occasion- by-instrument and
- residual (unexplained) error
When I take their main effects:
I come up with these as valid interactions, that agree with Bergeron and make sense to me as I have read Brennan and Shavelson:
- For classroom: cxo, cxi, cxoxi,e (Bergeron's residual).
- for occasion and method: cxo
So here is where I depart from Bergeron: the nested effects.
- for student(classroom) are these possible variance components?
- so:c; student by occasion, within classroom?
- is:c; student by student by instrument within classroom
- ro:c; rater by occasion, within classroom?
- ir:c; instrument by rater within classroom?
Also, why can't you have a sxo, rxo, sxi, and rxm interaction as variance components? Is it because the nesting essentially makes the sum of squares for the nested effect, put the interaction, in a fully crossed model. (e.g., sxo essentially=(s+sxc)xo because s:c equals s+sxc)? Therefore the nested effect there? or does it not work, because that would be examining an interaction of an interaction? and thats not really a thing?
I would really appreciate any resources on figuring valid variance components with nesting. Perhaps there is a valid variance components in nested designs for dummies or something that would hold my hand as I attempt to figure this out?
Bergeron, R., Floyd, R. G., McCormack, A. C., & Farmer, W. L. (2008). The Generalizability of Externalizing Behavior Composites and Subscale Scores across Time, Rater, and Instrument. School Psychology Review, 37(1), 91.