In my experiment participants answered to the same scale twice, after the presentation of two different stimuli, i.e., repeated measures design. The scale validation is not the main focus of my study, but since the scale is new, I would like to run an exploratory factor analysis on it, but of course now each participant answered to the same items twice.
I could run the same analysis twice, but since the results are very similar but not identical, it is hard to take decisions (e.g., eliminate or keep one item). Also, I have never seen a paper publishing two EFA of the same scale so it does not seem to be a common practice. Shell I consider only the first answer of the participant? Or rather the second one? What is the common practice?
I guess there might be some multilevel analysis that allows taking the repeated measure design into account, but honestly, since this is not the main focus of the study, I would go for a more simple solution if available.
Unfortunately, something is still unclear to me. If I understand it right, ICC compares each item scores at time 1 and time 2 and gives a sort of correlation score, so it is expected that the same item has similar scores at time 1 and time 2. However, since the experimental design presents two very different stimuli, the answers to the same items and times 1 and 2 are very different from each other, and, in my study, they are expected to be. So even though the structure of the scale is similar, comparing each item at t1 and t2 is going to give a poor ICC score. Is ICC going to be useful?
I'll try to be more explicit. I am using a scale to measure psychological need satisfaction. Each participant is presented with two different scenarios (order of presentation is randomized) and, after each scenario, the participant answers the need satisfaction scale. One scenario is meant to satisfy needs, the other one is meant not to satisfy needs. So participants usually give once low scores to the need satisfaction scale and once high scores. I want to present an EFA on the need satisfaction scale, i.e. all subjects are gonna have high scores on one administration and low scores on another one. The order of administration is randomized, but everyone has one with high scores and one with low scores. Can I show anyhow that the EFA is similar in both administrations?