2
$\begingroup$

I simulated some toy data of students taking a test. Each question in the test fell into one of four categories (which I called "concepts"), and each student had an "ability" for each of those four categories, which was a probability to get a question in that category right. The I used this setup to simulate a class of 250 students responding to a 30-question test.*

Then I ran the results through a factor analysis. As expected, I saw four large eigenvalues and 26 small ones on the scree plot. I repeated this with the same parameters except changing the number of categories, and as expected, the number of eigenvectors matched the number categories (and, after some rotation, the questions were mapped correctly to the categories as well).

Here are some scree plots of these results:

enter image description here

Up to four or five categories, there's a very clear drop-off in the eigenvalues, and by simulating a larger sample size, I can push this up to more categories.

My question is: how realistic is it to expect to find such clean results in any sort of real data? When I look at real test results I have on hand and do a factor analysis, the scree plots just die away basically smoothly. I can choose how many eigenvalues are significant based on some standardized criterion (e.g. eigenvalue >1), but in the example where I simulated the data, I know that I'm choosing the correct number of eigenvalues, and it's obvious. Is that ever realistically the case for actual data?

*student abilities in each category were chosen uniformly at random from 0 to 1

$\endgroup$
4
  • $\begingroup$ I think not. You can't extract lots of factors from a (relatively) small sample, and a (relatively) small number of items. Finding 9 factors in 30 items is unrealistic. $\endgroup$ – Jeremy Miles May 7 '18 at 17:25
  • $\begingroup$ Okay, but what about 3 or 4? $\endgroup$ – Mark Eichenlaub May 8 '18 at 1:02
  • $\begingroup$ Please remember that "scree-plot criterion" aka "Cattell's rule" is eigenvalues of the non-reduced matrix, that is, eigenvalues of PCA. It is one of several criterions to choose the number of factors. One should never rely only on one of them. $\endgroup$ – ttnphns May 8 '18 at 18:19
  • $\begingroup$ Clean-cut scree plot can be encountered in practice, why not?, especially of analyses of existing, well developed tests/questionnaires $\endgroup$ – ttnphns May 8 '18 at 18:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.