In my experiment participants answered to the same scale twice, after the presentation of two different stimuli, i.e., repeated measures design. The scale validation is not the main focus of my study, but since the scale is new, I would like to run an exploratory factor analysis on it, but of course now each participant answered to the same items twice.

I could run the same analysis twice, but since the results are very similar but not identical, it is hard to take decisions (e.g., eliminate or keep one item). Also, I have never seen a paper publishing two EFA of the same scale so it does not seem to be a common practice. Shell I consider only the first answer of the participant? Or rather the second one? What is the common practice?

I guess there might be some multilevel analysis that allows taking the repeated measure design into account, but honestly, since this is not the main focus of the study, I would go for a more simple solution if available.


Unfortunately, something is still unclear to me. If I understand it right, ICC compares each item scores at time 1 and time 2 and gives a sort of correlation score, so it is expected that the same item has similar scores at time 1 and time 2. However, since the experimental design presents two very different stimuli, the answers to the same items and times 1 and 2 are very different from each other, and, in my study, they are expected to be. So even though the structure of the scale is similar, comparing each item at t1 and t2 is going to give a poor ICC score. Is ICC going to be useful?

I'll try to be more explicit. I am using a scale to measure psychological need satisfaction. Each participant is presented with two different scenarios (order of presentation is randomized) and, after each scenario, the participant answers the need satisfaction scale. One scenario is meant to satisfy needs, the other one is meant not to satisfy needs. So participants usually give once low scores to the need satisfaction scale and once high scores. I want to present an EFA on the need satisfaction scale, i.e. all subjects are gonna have high scores on one administration and low scores on another one. The order of administration is randomized, but everyone has one with high scores and one with low scores. Can I show anyhow that the EFA is similar in both administrations?

  • 1
    $\begingroup$ You can't really do that with EFA alone, but CFA or SEM with appropriate constraints on loadings (and maybe residduals) could be used. This also implies that you know the factor structure beforehand. If you are interested in scale scores only, and you want to go for the simplest solution: use EFA on the first run, confirm factor loadings and factor structure on the second run using a second EFA (factor loadings should not deviate by more than a fixed value, say .05 or .1, between the two runs), and use an intraclass correlation between scale scores to assess score reliability. $\endgroup$ – chl Oct 21 '20 at 15:37
  • $\begingroup$ Thank you for your fast answer. I created the scale to be composed of one-factor and preliminary EFAs agree on this. So, if I understand right, I should run two EFAs (using a threshold to check that fator loadings are stable enough), but then what should I publish? Both EFAs? The intraclass correlation should replace the Chronbach's alpha? Sorry for my many questions... $\endgroup$ – Silvia Oct 21 '20 at 15:46
  • 1
    $\begingroup$ If you already validated your unidimensional scale using preliminary EFAs, then a structural equation model that accounts for repeated measurements would allow to incorporate measurement error, and it's the best approach. Otherwise, anything that works on numerical scores would do the job (at the cost of ignoring measurement error and measurement invariance). It all depends on what you're primarily interested in: scores reliability (ICC) or scale structure (SEM). The main problem lies in the possible learning effect, or other form of bias, that could affect individual responses. $\endgroup$ – chl Oct 21 '20 at 15:53
  • $\begingroup$ My previous answer was not clear. It is the first time I use this scale and the preliminary EFAs I mentioned are the two that I run on this data, one EFA on items for the first stimulus and the second EFA on items for the second stimulus. They are not identical, but they identify a mono-factorial structure, as I expect. Now my question is what to put in the related paper. $\endgroup$ – Silvia Oct 21 '20 at 15:59
  • 1
    $\begingroup$ Then report the results of the first EFA, with factor loadings of the seocnd EFA. However, this does not ensure that measurement accurately reflects individual latent scores. CFA and SEMs are cheap nowadays (both Stata and R have facilities for that, as well as Mplus). You can then use latent scores as the main outcomes in your next analyses. $\endgroup$ – chl Oct 21 '20 at 16:43

To sum up my comments, EFA suffers from two main drawbacks in this context:

  • It doesn't incorporate the design effect (repeated measures, which implies a specific variance-covariance matrix);
  • There's no way to tell whether one model is better than the other: There's no real goodness-of-fit measure in EFA.

If you are only interested in scale reliability, then computing the intraclass correlation (with its associated 95% confidence interval) should be enough. This won't, however, take into account measurement error unless you correct for it, as suggested by William Revelle on his Personality Project.

If the factor structure matters, the proper way to analyze such data would be to use some Structural Equation Model (or CFA under the umbrella of the multilevel factor analysis) since this allows to account for repeated measures. One of the benefit of this approach is that you can directly use the factor scores (i.e., accounting for incorporate measurement error) for further processing at no cost (regression, comparison of means, etc.).

In response to comments, the above suggestion assumes that items (item content + response options) are constant in your scale, i.e. items do not vary from one administration to the other (like, e.g., when using different images which all relate to the same category, and subjects are asked to rate them).

If, on the contrary, the pre-post scores aren't necessarily related on a per subject basis (or they are simply be anti-correlated), you're probably more interested in demonstrating that the interitem correlation matrix is comparable between the two administrations (since this will account for the intra-individual high/low balance), that item loadings are close one each other, and that the scale has adequate internal consistency (Cronbach alpha or other related indices). Since EFA is mainly concerned with factor structure and interitem correlation, it's probably the best way to go. It will be harder to work directly with raw or factor scores, unless you standardize them (using reverse scoring or other kind of absolute transformation) so that they remain comparable from one administration to the other, but analyzing the observed correlation matrix (between items, for all subjects) using classical data analysis techniques (PCA, MCA or cluster analysis) should be enough.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.