We have a survey instrument and are interested in assessing dimensionality of it. Looking at plots of multidimensional scaling, it appears as though there are, perhaps, 3 distinct dimensions to the survey since there are 3 seemingly well defined clusters of responses.

When I perform a Scree plot, I observe there are 7 dimensions for which the eigenvalues are greater than one, so the Kaiser-Guttman indicates this anticonservative high dimensionality. Doing a parallel analysis by generating independent random normal values of the same shape as the original matrix of responses gives a much more conservative 4 dimensions before eigenvalues become consistent with those of randomly generated data.

However, the parallel analysis doesn't factor in the coding and format of the data, and differing distributions of possible covariances under the null hypothesis. It makes sense to do a permutation test of the original response data by permuting values in each column. This maintains the same mean and standard deviation for these values, but finds a sampling distribution for the correlation matrix under a null hypothesis of completely independent data.

Has this been explored before? Is there a name for such a test? Are there caveats to this approach?

  • $\begingroup$ It seems to me that you ought to give more elaborate account to the usual "parallel analysis" approach and to what makes you dissatisfied so that you cenceive yet another permutation procedure. Currently, the parallel analysis doesn't factor in the coding and format of the data, and differing distributions of possible covariances under the null hypothesis. is too short and a bit mystic to understand. $\endgroup$
    – ttnphns
    Feb 7 '14 at 6:48
  • $\begingroup$ @ttnphns This article may provide some of the missing details. PA is not a permutation procedure, since you're not sampling from the empirical distribution function, but from multivariate normal data. $\endgroup$
    – AdamO
    Feb 7 '14 at 15:21

Several authors have explored rerandomization-based criteria for component retention. For example, Peres-Neto et al., (2009) implemented a rerandomization-based version of parallel analysis and of Monte Carlo parallel analysis (which uses a high centile of eigenvalues of random data, rather than the mean). Dray developed a rerandomization-based component retention method using the RV-coefficient.

Aside: I use the term rerandomization-based to indicate the process by which uncorrelated comparator data sets, such as those used in parallel analysis, are generated. One can preserve the exact univariate distribution of each variable in observed data $\mathbf{X}$, while reducing correlation with each other variable in $\mathbf{X}$ to chance by independently rerandomizing (i.e. shuffling, or sampling without replacement all observations) each variable .

However, Dinno (2009) found that parallel analysis was insensitive to the distributional form of the data. This is because the (usual) object of analysis in principal component analysis is the correlation matrix, $\mathbf{R}$, and with any decent sample size, linear correlations become accurate regardless of the distribution of the data.


Dinno, A. (2009). Exploring the Sensitivity of Horn’s Parallel Analysis to the Distributional Form of Simulated Data. Multivariate Behavioral Research, 44(3):362–388.

Dray, S. (2008). On the number of principal components: A test of dimensionality based on measurements of similarity between matrices. Computational Statistics & Data Analysis, 52(4):2228–2237.

Peres-Neto, P., Jackson, D., and Somers, K. (2005). How many principal components? stopping rules for determining the number of non-trivial axes revisited. Computational Statistics & Data Analysis, 49(4):974–997.

  • $\begingroup$ This agrees with my back-of-the-envelope simulation findings. Thanks for the thorough answers! $\endgroup$
    – AdamO
    Apr 23 '14 at 19:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.