So, I ran an EFA on 60 items. Analysis resulted in 19 components with an eigenvalue of a score greater than 1. The only factors that theoretically make sense and that include more then 3 items have eigenvalues greater than 3 - can I use these first three components in my analysis or do I have to rerun the analysis?
This is not a stupid question, in fact its a very good question.
To quote William Revelle (author of the R package
psych) who quotes Kaiser saying "Solving the number of factors problem is easy, I do it everyday before breakfast. But knowing the right solution is harder".
The trouble with factor solutions is not that there are no criteria, it is that they often disagree. For example, parallel analysis, the VSS criterion (developed by Revelle linked above) and the scree test, as well as the eigenvalues greater than one typically suggest different numbers of factors, and it is difficult to know which one is "correct".
Typically (if I have enough data), I will split the dataset into a number of smaller pieces and apply multiple criteria to each subset (scree test, parallel analyses et al). I then apply the models developed to some of the other subsets, and test them using structural equation modelling (i.e. confirmatory factor analysis) to decide which one performs better on new data.
However, even after all that, the "best" in a statistical model sense is sometimes a model which even I don't believe. So, its a difficult problem.
There is some work in chemometrics which looks at this problem from a cross-validation perspective, a good (if mathematically dense) review can be found here.
So, to answer your original question, yes you can use five or (ten) factors to model your dataset, but you will have to justify your choice.
Additionally, you should ensure that you have tried both orthogonal and oblique rotations (as this may help to illuminate structure). It is also possible that some of the items are not a good fit for the scale you are using, and if you examine Cronbach's alpha for reliability you may be able to remove some ill-fitting items. Tabachnick and Fidell note that factors with three or less items are typically unstable so you may be able to eliminate some of them.
Some useful criteria and implementations in R for choosing the correct number of factors can be found in the
psych package, the
nFactors package, as well as the
bcd package (which implements the Wold and Gabriel methods for choosing the rank of a matrix which are reviews in the paper I referenced above). I hope this helps.