4
$\begingroup$

How can we trust the significance tests for any of our factor loadings, or even the correlations matrix, when so many tests are run in factor analysis?

Especially for IRT models where we may be analyzing multiple items in a scale.

$\endgroup$
2
$\begingroup$

As I understand you can't trust $p$ values in significance tests of the factor loadings, though researchers mostly ignore this in practice. Whenever we have many hypotheses being tested simultaneously, the inflation of Type I error rate may be an issue. This question Multiple hypothesis testing in SEM deals with a similar issue in the context of structural equation modeling. In the article linked in the answer to that question, it is argued that confirmatory factor analyses should use a procedure for reducing Type I errors and that using false discovery rate procedures offers a good balance between Type I and Type II error rates.

$\endgroup$
6
  • 1
    $\begingroup$ Do you have a reference for the claim that p values for factor loadings are not trustworthy? Though I agree researchers typically ignore them (and to say so would be mostly opinion-based at this point), it seems another thing entirely to say that the p values are inaccurate. $\endgroup$ – jsakaluk Jan 31 '18 at 4:36
  • 1
    $\begingroup$ @jsakaluk The reference is in the liked question, I will change the answer to list it directly. $\endgroup$ – Chris Novak Jan 31 '18 at 9:13
  • $\begingroup$ Thank you both! Is there any evidence to suggest that measurement models (or even structural models) are robust to multiple comparisons, in the same way that omnibus F-tests (weakly) protects against multiplicity? $\endgroup$ – PyjamaNinja Jan 31 '18 at 18:40
  • 1
    $\begingroup$ @PyjamaNinja Not exactly, but the omnibus F-test analogue in CFA and structural models is the chi-squared test, and the fit indices (RMSE, etc.) perform a similar role. In practice they are there to evaluate the whole model simultaniously, and only later the individual loadings are examined. $\endgroup$ – Chris Novak Jan 31 '18 at 22:55
  • 1
    $\begingroup$ @PyjamaNinja I agree they do not protect against multiple testing because they are used for assessing misspecification rather than testing the joint effects as a proper omnibus test would. $\endgroup$ – Chris Novak Feb 3 '18 at 16:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.