Unexpected factor structure in different samples

Question

I have administered a questionnaire to respondents asking about customers' adoption of technology. I want to do an exploratory principal components factor analysis in Stata to form constructs from a couple of questions (in line with the existing literature).

The variables that I want to construct are perceived usefulness, perceived ease of use, perceived risk with product (PRP), etc. and I am comparing customers from two countries: The Netherlands (n=63) and Bulgaria (n=101).

I designed items based on the research literature. So I expected that there would be no problems with the factor analysis; i.e., all items would have high factor loadings on a single factor. I have obtained cronbach's alpha which is 0.7 for all factors and 0.6 for usefulness for the Dutch customers, which is OK. The factor analysis is fine for most of the factors and most of items load most on one single factor as suggested by literature.

The problem is that for PRP I got 4/5 factors for Bulgarians and 6/7 factors for Dutch which are actually different constructs compared to PRP for Bulgarians (13 factor loadings for PRP). I also have similar problem with Subjective norm (SN).

My questions:

• When items do not fit into one factor is it possible to just take the means of such items? *
• Or is there something else I should do?
• Can you also provide me with some reference?

Answer 1

As a general advice, your sample size is quite small. It's not such a no-no as some people claim but depending on the specifics of the data, it's not too surprising to have unstable or unexpected results in a factor analysis.

A big question in all this is how you selected the number of factors to extract. There is no objective easy-to-determine “number of factors” in the data, so when you are saying “I have got 4/5 factors” there could be a lot of reasons for that. The default choice in some software packages (e.g. SPSS) have been heavily criticized.

If after reading up on that and adjusting the analysis, you are still unable to recover a reasonable structure, what you should do next depends mostly on the source of your scale:

• If this is an ad hoc scale that you created yourself, it is not surprising at all that some items do not perform as expected even if you formulated them based on a well-established literature. Ideally, you should collect a larger sample and try to refine the scale. This is a large and complex topic so it's difficult to give specific advice without a more focused question but it is common to start with a large set of items and reduce it using factor analysis. You could therefore just focus on the most interpretable factors and select the items that seem most cleanly related to them, throwing the rest out. Of course, these shorter scales will have a lower reliability than if all the items worked as well as intended.

• If this is a translation of a well-known scale, this is worrisome and there is no easy solution. Ideally, you would review the translation item-by-item and collect a much larger sample size for a validation study (perhaps with some confirmatory factor analysis). Again, this is a large and complex topic so it's difficult to give more specific advice.

• If this is well-known questionnaire that has already been used in Dutch and Bulgarian samples, you could just ignore the EFA results and go on with your analysis (you could in fact argue that there was no point in trying an exploratory factor analysis of a well-known scale with such a tiny sample in the first place).

Also, you could try to look at the specific item loadings on each factor. Intuitively, a messy structure with items “switching” factors is more of a problem than factors split in half with all items supposed to measure a construct ultimately falling together in two or more subfactors.

Answer 2

What comes first here? Understanding the data or being committed to using a particular technique? Why precisely do you think the PCA is going to help?

Perhaps it is beyond your control, or too late any way, but the questionnaire sounds badly designed. If a rating of "perceived usefulness" is needed, why not ask something like "on a scale of 1 to 5, please rate how useful this thing is"? (Similar comments for other scales.)

More positively, I would

1. pool the data for two countries, and then see how the people from each country compare on the scales defined by the PCs. The different sample sizes don't seem especially worrying here.

2. do a PCA for Netherlands, then see (as above).

3. do a PCA for Bulgaria, then see (as above).

4. see which of #1 to #3 was most illuminating.

I wouldn't try anything more complicated. What you have in mind sounds like an analysis where you would have a hard time disentangling side-effects of what you did from something revealing the structure of the data.

I am calling what you did PCA, on the philosophy of calling a spade a spade (you may need a different idiom in your first language).