How to do exploratory factor analysis in a repeated measure design?

Question

In my experiment participants answered to the same scale twice, after the presentation of two different stimuli, i.e., repeated measures design. The scale validation is not the main focus of my study, but since the scale is new, I would like to run an exploratory factor analysis on it, but of course now each participant answered to the same items twice.

I could run the same analysis twice, but since the results are very similar but not identical, it is hard to take decisions (e.g., eliminate or keep one item). Also, I have never seen a paper publishing two EFA of the same scale so it does not seem to be a common practice. Shell I consider only the first answer of the participant? Or rather the second one? What is the common practice?

I guess there might be some multilevel analysis that allows taking the repeated measure design into account, but honestly, since this is not the main focus of the study, I would go for a more simple solution if available.

---

Edits

Unfortunately, something is still unclear to me. If I understand it right, ICC compares each item scores at time 1 and time 2 and gives a sort of correlation score, so it is expected that the same item has similar scores at time 1 and time 2. However, since the experimental design presents two very different stimuli, the answers to the same items and times 1 and 2 are very different from each other, and, in my study, they are expected to be. So even though the structure of the scale is similar, comparing each item at t1 and t2 is going to give a poor ICC score. Is ICC going to be useful?

I'll try to be more explicit. I am using a scale to measure psychological need satisfaction. Each participant is presented with two different scenarios (order of presentation is randomized) and, after each scenario, the participant answers the need satisfaction scale. One scenario is meant to satisfy needs, the other one is meant not to satisfy needs. So participants usually give once low scores to the need satisfaction scale and once high scores. I want to present an EFA on the need satisfaction scale, i.e. all subjects are gonna have high scores on one administration and low scores on another one. The order of administration is randomized, but everyone has one with high scores and one with low scores. Can I show anyhow that the EFA is similar in both administrations?

Answer 1

To sum up my comments, EFA suffers from two main drawbacks in this context:

• It doesn't incorporate the design effect (repeated measures, which implies a specific variance-covariance matrix);
• There's no way to tell whether one model is better than the other: There's no real goodness-of-fit measure in EFA.

If you are only interested in scale reliability, then computing the intraclass correlation (with its associated 95% confidence interval) should be enough. This won't, however, take into account measurement error unless you correct for it, as suggested by William Revelle on his Personality Project.

If the factor structure matters, the proper way to analyze such data would be to use some Structural Equation Model (or CFA under the umbrella of the multilevel factor analysis) since this allows to account for repeated measures. One of the benefit of this approach is that you can directly use the factor scores (i.e., accounting for incorporate measurement error) for further processing at no cost (regression, comparison of means, etc.).

---

In response to comments, the above suggestion assumes that items (item content + response options) are constant in your scale, i.e. items do not vary from one administration to the other (like, e.g., when using different images which all relate to the same category, and subjects are asked to rate them).

If, on the contrary, the pre-post scores aren't necessarily related on a per subject basis (or they are simply be anti-correlated), you're probably more interested in demonstrating that the interitem correlation matrix is comparable between the two administrations (since this will account for the intra-individual high/low balance), that item loadings are close one each other, and that the scale has adequate internal consistency (Cronbach alpha or other related indices). Since EFA is mainly concerned with factor structure and interitem correlation, it's probably the best way to go. It will be harder to work directly with raw or factor scores, unless you standardize them (using reverse scoring or other kind of absolute transformation) so that they remain comparable from one administration to the other, but analyzing the observed correlation matrix (between items, for all subjects) using classical data analysis techniques (PCA, MCA or cluster analysis) should be enough.