0
$\begingroup$

I have a scale that I would like to validate using CFA. The scale measures different aspects of fear of any given stimulus. In my first study, I had participants create their own stimulus and then they responded to the scale items. This means that each participant had a different stimulus from the others. EFA suggested that there are two factors for my scale.

For my validation study, I would like to not only confirm the factor structure but also to validate each factor using experimental manipulation. For this, I have developed stimuli with a 2 x 2 x 2 design, addressing either factor 1 (high vs. low) or factor 2 (high vs. low) with two different forms of presentation (A vs. B). I would prefer a within subjects manipulation so that the total number of participants can be reduced.

My main interest is in showing that the assumed factor structure can be confirmed (for each of the 8 stimuli and in general) and that the two factors are sensitive to their respective experimental manipulation. Additionally, it might be interesting to test measurement invariance across stimuli.

What would be the proper way to answer these questions and which sample size requirements are associated with the respective approach?

In the literature, I found these options:

  1. treat the observations as independent (ignoring the within-subjects design), using multi-group analysis to test for measurement invariance (would I have 8 groups then?) and including manifest, binary factors for the experimental manipulations Sketch for option 1
  2. model the correlated residuals by hand Sketch for option 2
  3. multilevel CFA
  4. create individual models for each experimental cell Sketch for option 4

Note that all my variables are on level 1, therefore option 1 does seem quite attractive to me and option 3 seems to be overly complex since there are no level 2 or cross-level associations that I would like to model. Option 2 would result in a huge model and I doubt that this would require a manageable sample size. Option 4 would also seem interesting, however, it would not allow me to test measurement invariance, right?

So, what would be the most correct approach and which approach would be still acceptable but more feasible? Maybe a combination of 1 and 4? Are there other options that I am missing?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ How many subjects have repeatedly been given the survey? $\endgroup$
    – Erik Ruzek
    Commented Nov 20, 2022 at 23:00
  • $\begingroup$ @ErikRuzek This is still up for debate. I thought of 300 subjects but I could do more if it is necessary. $\endgroup$
    – mkks
    Commented Nov 21, 2022 at 9:58
  • 1
    $\begingroup$ That's helpful. Keep in mind that SEM requires large samples, so 300 is probably at the lower limit of sample size for the complexity of models you are considering. Have you considered using the MIMIC framework, which can be useful with smaller sample sizes and can test invariance in addition to mean differences in the latent factor(s) across groups? See ncbi.nlm.nih.gov/pmc/articles/PMC5140785 And on CV stats.stackexchange.com/questions/554687/… $\endgroup$
    – Erik Ruzek
    Commented Nov 21, 2022 at 18:32
  • $\begingroup$ @ErikRuzek Thank you very much for the links. I would try to get 400-500 subjects then. As far as I understand, the MIMIC model would be an alternative to multi-group cfa. But would it also help with my repeated measurement problem? Would you agree that it would be acceptable to treat the observations as independent? Also, I am not sure the MIMIC model differs from the model I sketched in option 1 $\endgroup$
    – mkks
    Commented Nov 23, 2022 at 9:06

1 Answer 1

Reset to default
0
$\begingroup$

In the lavaan forum I was pointed towards a further solution: cluster-robust standard errors. It allows fitting a flat model (without between-participant effects) but still accounts for the hierarchical nature of the data. However, they are only suited, as far as I have understood, in the case where all variables in the model are on level 1. If that is the case, cluster-robust standard errors can be utilized in lavaan by specifying the cluster argument in the cfa()-function; e.g.: cfa(model, data = df, cluster = "participant_id"). The model syntax does not differ from that of a regular flat cfa model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.