Next steps after performing a principal component analysis?

A few months ago, I developed a questionnaire using a principal component analysis (PCA) and tested the questionnaire for split-half reliability (using a sample which I will call sample #1).

I am in the process of writing a manuscript to submit for publication, which utilizes the questionnaire and its relationship to depression, anxiety, and stress. I have never developed a questionnaire before, and I don't know what further analyses to conduct with the new sample (sample #2). It should be noted that sample #2 is different from sample #1 but not terribly so (both samples are a composite of undergraduate students and individuals who attend exercise classes).

I already did the split-half reliability for the questionnaire with sample #2 and it was above .8, but is there anything else that should be done? Do I need to somehow confirm the factor structure of the questionnaire?

I did a PCA with sample #2 just to see what would happen, but the items are not lining up with the factors that were specified when I first did the PCA. Frankly, I want to know if I made a mistake in analyzing the questionnaire data for sample #2 with a PCA.

Should I have done the second principal components analysis, or is another analysis "better"? Or, should I just stick with the split-half reliability?

-
How have you used PCA to initially developing the questionnaire? Have you used the PCA to choose the better items and discard the worst? –  this.is.not.a.nick Jul 20 '12 at 15:23
Yes, I used PCA to initially develop the questionnaire. We used it to choose the better items and discard the worst. We also used it to identify 4 factors. Do I need to do anything after the PCA? I just read in an article that you have to do an EPA with a new sample and then a CFA - is this true? –  Madeline Jul 22 '12 at 12:43

Principal component analysis just presents the orthogonal linear combinations of your variables that explain the most variability in your data. If you apply it to two mutlivariate data sets even if they are similar they will give different answers just due to random variability unless the sample size is very large and they really do both come from nearly identical multivariate populations. Also when I say very large that depends somewhat on the dimensionality. The large the number of variables the more samples you need. Par of the problem is the curse of dimensionality which state that the higher the number of dimnsions the more the data will spread out away from the center.

Principal component analysis could be used as a tool in regression analysis, clustering or classification problems because it is basically a dimension reduction technique as it often shows that most of the variability in the data can be explained by the first few principal components. So a good characterization of the data can be seen in lower dimensional spaces. Projection of the data on the first two principal components can be very useful in identifying characteristics in the data. It is basically an exploratory data analysis tool.

I am not familiar with split-half reliability and so I don't know how you are trying to use principal components there. What you should do next really depends on what your objective is? Maybe you should look at scatter plots of both samples on the first 2 prinicpal components found from the first sample. If they spread out differently that at least tells you something about how the samples differ and perhaps why the second sample had different principal components. If they look very similar in the scatter plot it might confirm that the two sample do come from the same poulation and that randomness in the data is all that is causing some shifting of the prinicipal components. Remember that each principal component is obtained by maximizing the variance in that data set in a direction orthogonal to the principal components that came before it and the first principal component is obtained by searching for the direction in the k dimensional space that explains the highest percentage of variation in the data set.

Given that all this optimization is going on relative to a particular data set it should not be surprising that the two data sets produce different principal components. In some situations the solution may be to combine the two samples and get the principal compoents for the combined data set to use in the future analysis. In other cases the differences can be so great that you would want to rethink whether or not the data really are similar.

-
Thank you for your answer. The main objective that I am trying to accomplish is to create a questionnaire which has a confirmed or at least identified factor structure. I read in an article just now that you should use PCA for data reduction, then EFA (exploratory factor analysis) to identify factors, then CFA (confirmatory factor analysis) to confirm the factor structures. To your best knowledge, is this the case? –  Madeline Jul 22 '12 at 12:50
So far I have conducted 2 PCAs: First PCA to reduce # of items and create factors, then Second PCA to confirm factor structure (with factors obviously not matching up correctly). Was my method the right way to confirm the factor structure, or did I use PCA incorrectly? (P.S. In psychometrics, split-half reliability is used to ensure that participant responses are consistent between the first and second half of a data set.) –  Madeline Jul 22 '12 at 12:51
I will also try looking at the scatter plots - thank you for suggesting that approach –  Madeline Jul 22 '12 at 12:52