A few months ago, I developed a questionnaire using a principal component analysis (PCA) and tested the questionnaire for split-half reliability (using a sample which I will call
I am in the process of writing a manuscript to submit for publication, which utilizes the questionnaire and its relationship to depression, anxiety, and stress. I have never developed a questionnaire before, and I don't know what further analyses to conduct with the new sample (
sample #2). It should be noted that sample #2 is different from sample #1 but not terribly so (both samples are a composite of undergraduate students and individuals who attend exercise classes).
I already did the split-half reliability for the questionnaire with sample #2 and it was above .8, but is there anything else that should be done? Do I need to somehow confirm the factor structure of the questionnaire?
I did a PCA with sample #2 just to see what would happen, but the items are not lining up with the factors that were specified when I first did the PCA. Frankly, I want to know if I made a mistake in analyzing the questionnaire data for sample #2 with a PCA.
Should I have done the second principal components analysis, or is another analysis "better"? Or, should I just stick with the split-half reliability?