We have a survey instrument and are interested in assessing dimensionality of it. Looking at plots of multidimensional scaling, it appears as though there are, perhaps, 3 distinct dimensions to the survey since there are 3 seemingly well defined clusters of responses.
When I perform a Scree plot, I observe there are 7 dimensions for which the eigenvalues are greater than one, so the Kaiser-Guttman indicates this anticonservative high dimensionality. Doing a parallel analysis by generating independent random normal values of the same shape as the original matrix of responses gives a much more conservative 4 dimensions before eigenvalues become consistent with those of randomly generated data.
However, the parallel analysis doesn't factor in the coding and format of the data, and differing distributions of possible covariances under the null hypothesis. It makes sense to do a permutation test of the original response data by permuting values in each column. This maintains the same mean and standard deviation for these values, but finds a sampling distribution for the correlation matrix under a null hypothesis of completely independent data.
Has this been explored before? Is there a name for such a test? Are there caveats to this approach?