The valid variance components to include in a generalizability theory model (anova) involving nesting? Are these correct interactions?

Question

My question is about generalizability theory. A study by Bergeron, Floyd, McCormack, and Farmer (2008) is only adding to this confusion. Basically, in G-theory you want the first model you run to include all main effects and valid interactions. This is fairly straightforward for a crossed design, but nesting makes this more difficult for me to understand. I was reading Bergeron et al., 2008, and this is adding to the confusion.

They stated, on page 97,

We used a three-facet, partially nested design, with students and raters nested within classrooms and classrooms crossed with occasions and instruments. Variance components were computed for five main sources of variation (classroom, student-within-classroom, rater-within-classroom, occasion, and instrument), four two-way interactions (student-by-rater-within-classroom, classroom-by-occasion, classroom-by-instrument, and occasion-by-instrument), and residual (unexplained) error.

1. First, it sounds like they have five facets (factors), not three: student, rater, classroom, occasion, and instruments. Is that correct? or because the student and rater are nested, then you cannot consider them a true facet?

2. Second, the variance components in their model aren't all valid possible combinations, I think they are missing some?

   • Bergeron main effects as reported:
   • classroom,
   • student(classroom),
   • rater(classroom),
   • occasion,
   • instrument.
   • Bergeron second level effects as reported:
   • student-by-rater-within-classroom,
   • classroom-by-occasion,
   • classroom-by-instrument
   • occasion- by-instrument and
   • residual (unexplained) error

   When I take their main effects:

   • classroom,
   • student(classroom),
   • rater(classroom),
   • occasion,
   • instrument.

   I come up with these as valid interactions, that agree with Bergeron and make sense to me as I have read Brennan and Shavelson:

   • For classroom: cxo, cxi, cxoxi,e (Bergeron's residual).
   • for occasion and method: cxo
   • sxr(classroom)

   So here is where I depart from Bergeron: the nested effects.

   • for student(classroom) are these possible variance components?
   • so:c; student by occasion, within classroom?
   • is:c; student by student by instrument within classroom
   • ro:c; rater by occasion, within classroom?
   • ir:c; instrument by rater within classroom?

3. Also, why can't you have a sxo, rxo, sxi, and rxm interaction as variance components? Is it because the nesting essentially makes the sum of squares for the nested effect, put the interaction, in a fully crossed model. (e.g., sxo essentially=(s+sxc)xo because s:c equals s+sxc)? Therefore the nested effect there? or does it not work, because that would be examining an interaction of an interaction? and thats not really a thing?

4. I would really appreciate any resources on figuring valid variance components with nesting. Perhaps there is a valid variance components in nested designs for dummies or something that would hold my hand as I attempt to figure this out?

Bergeron, R., Floyd, R. G., McCormack, A. C., & Farmer, W. L. (2008). The Generalizability of Externalizing Behavior Composites and Subscale Scores across Time, Rater, and Instrument. School Psychology Review, 37(1), 91.

Answer 1

I would strongly recommend the 2012 article by Bloch.(1) It will probably help you greatly. As far as the number of facets, G-theory always has 1 less than say IRT or Rasch. For example, candidate ability might be the measure of interest, and the factors (or facets) that a researcher is interested in might be: rater, cases, track, etc. Candidate is not considered a facet/factor.

Your questions are valid questions; please read the article and follow its references.

1. Boch R, Norman GR. Generalizability theory for the perplexed: A practical introduction and guide: AMEE Guide No. 68. Med Teach. 2012;34(11):960–92.

Answer 2

First of all, yes, the design consists of 5 facets and not 3.

The rule about which interactions can or cannot be estimated in a design that involves nesting is the following: all combinations of factors are allowed except those that involve an interaction between two nested factors.

In this design, that means SxC and RxC interactions, and any higher-order interactions involving these, are disallowed. So in addition to the 4 two-way interactions given by the authors, it is also possible to estimate effects for:

• SxO: Students are rated on multiple Occasions
• RxO: Raters give ratings on multiple Occasions
• SxI: Students are rated on multiple Instruments
• RxI: Raters give ratings using multiple Instruments

We also have the following three-way interactions:

• SxRxO: SxR interaction effects are systematically different on different Occasions
• SxRxI: etc...
• SxOxI
• RxOxI
• CxOxI

And finally, the residual term is equivalent to (that is, confounded with) the four-way SxRxOxI interaction.

As for why certain interactions cannot be estimated in designs that involve nesting -- for example, in the present design we cannot estimate Student-by-Classroom (SxC) interactions -- it helps to think about the definition of an interaction. For example, an SxC interaction means that the Student effect "depends on" the Class factor, in the sense that the Student effect is different in different classrooms. So maybe a particular student performs very poorly in one classroom, but if they were in another classroom they would perform much better. The problem is that in order to estimate this empirically, we have to actually observe the student in both (and other) classrooms. But if Students are nested in Classrooms, then by definition we observe each student in one and only classroom. There very well may be SxC interaction effects in our data, but they are impossible to estimate in this kind of design.

And then if the SxC interactions can't be estimated, then higher-order interactions involving SxC -- for example, SxCxO effects, which would hold that the magnitude of the SxC interaction is different on different Occasions -- can't be estimated either, because in order to see if SxC effects differ as as function of Occasion, we obviously have to be able to estimate the SxC effects in the first place!