Permutation test for factor analysis

Question

We have a survey instrument and are interested in assessing dimensionality of it. Looking at plots of multidimensional scaling, it appears as though there are, perhaps, 3 distinct dimensions to the survey since there are 3 seemingly well defined clusters of responses.

When I perform a Scree plot, I observe there are 7 dimensions for which the eigenvalues are greater than one, so the Kaiser-Guttman indicates this anticonservative high dimensionality. Doing a parallel analysis by generating independent random normal values of the same shape as the original matrix of responses gives a much more conservative 4 dimensions before eigenvalues become consistent with those of randomly generated data.

However, the parallel analysis doesn't factor in the coding and format of the data, and differing distributions of possible covariances under the null hypothesis. It makes sense to do a permutation test of the original response data by permuting values in each column. This maintains the same mean and standard deviation for these values, but finds a sampling distribution for the correlation matrix under a null hypothesis of completely independent data.

Has this been explored before? Is there a name for such a test? Are there caveats to this approach?

Answer 1

Several authors have explored rerandomization-based criteria for component retention. For example, Peres-Neto et al., (2009) implemented a rerandomization-based version of parallel analysis and of Monte Carlo parallel analysis (which uses a high centile of eigenvalues of random data, rather than the mean). Dray developed a rerandomization-based component retention method using the RV-coefficient.

Aside: I use the term rerandomization-based to indicate the process by which uncorrelated comparator data sets, such as those used in parallel analysis, are generated. One can preserve the exact univariate distribution of each variable in observed data $\mathbf{X}$, while reducing correlation with each other variable in $\mathbf{X}$ to chance by independently rerandomizing (i.e. shuffling, or sampling without replacement all observations) each variable .

However, Dinno (2009) found that parallel analysis was insensitive to the distributional form of the data. This is because the (usual) object of analysis in principal component analysis is the correlation matrix, $\mathbf{R}$, and with any decent sample size, linear correlations become accurate regardless of the distribution of the data.

References

Dinno, A. (2009). Exploring the Sensitivity of Horn’s Parallel Analysis to the Distributional Form of Simulated Data. Multivariate Behavioral Research, 44(3):362–388.

Dray, S. (2008). On the number of principal components: A test of dimensionality based on measurements of similarity between matrices. Computational Statistics & Data Analysis, 52(4):2228–2237.

Peres-Neto, P., Jackson, D., and Somers, K. (2005). How many principal components? stopping rules for determining the number of non-trivial axes revisited. Computational Statistics & Data Analysis, 49(4):974–997.