I have a scale that I would like to validate using CFA. The scale measures different aspects of fear of any given stimulus. In my first study, I had participants create their own stimulus and then they responded to the scale items. This means that each participant had a different stimulus from the others. EFA suggested that there are two factors for my scale.
For my validation study, I would like to not only confirm the factor structure but also to validate each factor using experimental manipulation. For this, I have developed stimuli with a 2 x 2 x 2 design, addressing either factor 1 (high vs. low) or factor 2 (high vs. low) with two different forms of presentation (A vs. B). I would prefer a within subjects manipulation so that the total number of participants can be reduced.
My main interest is in showing that the assumed factor structure can be confirmed (for each of the 8 stimuli and in general) and that the two factors are sensitive to their respective experimental manipulation. Additionally, it might be interesting to test measurement invariance across stimuli.
What would be the proper way to answer these questions and which sample size requirements are associated with the respective approach?
In the literature, I found these options:
- treat the observations as independent (ignoring the within-subjects design), using multi-group analysis to test for measurement invariance (would I have 8 groups then?) and including manifest, binary factors for the experimental manipulations Sketch for option 1
- model the correlated residuals by hand Sketch for option 2
- multilevel CFA
- create individual models for each experimental cell Sketch for option 4
Note that all my variables are on level 1, therefore option 1 does seem quite attractive to me and option 3 seems to be overly complex since there are no level 2 or cross-level associations that I would like to model. Option 2 would result in a huge model and I doubt that this would require a manageable sample size. Option 4 would also seem interesting, however, it would not allow me to test measurement invariance, right?
So, what would be the most correct approach and which approach would be still acceptable but more feasible? Maybe a combination of 1 and 4? Are there other options that I am missing?