I have a matrix of 1000 observations and 50 variables each measured on a 5-point scale. These variables are organized into groups, but there aren't an equal number of variables in each group.

I'd like to calculate two types of correlations:

  1. Correlation within groups of variables (among characteristics): some measure of whether the variables within the group of variables are measuring the same thing.
  2. Correlation between groups of variables: some measure, assuming that each group reflects one overall trait, of how each trait (group) is related to every other trait.

These characteristics have been previously classified into groups. I'm interested in finding the correlation between the groups - i.e. assuming that the characteristics within in group are measuring the same underlying trait (having completed #1 above - Cronbach's alpha), are the traits themselves related?

Does anybody have suggestions for where to start?

  • 1
    $\begingroup$ If you are familiar with R, there is a package vegan with functions anosim or preferably, adonis (permutational MANOVA). $\endgroup$ Apr 24, 2011 at 5:15
  • $\begingroup$ I've updated your question to try to use standard terminology (i.e., variables not characteristics; groups of variables rather than "groups") $\endgroup$ Apr 25, 2011 at 5:28

4 Answers 4


What @rolando suggested looks like a good start, if not the whole response (IMO). Let me continue with the correlational approach, following the Classical Test Theory (CTT) framework. Here, as noted by @Jeromy, a summary measure for your group of characteristics might be considered as the totalled (or sum) score of all items (a characteristic, in your words) belonging to what I will now refer to as a scale. Under CTT, this allows us to formalize individual "trait" propensity or liability as one's location on a continuous scale reflecting an underlying construct (a latent trait), although here it is merely an ordinal scale (but this another debate in the psychometrics literature).

What you described has to do with what is know as convergent (to what extent items belonging to the same scale do correlate one with each other) and discriminant (items belonging to different scales should not correlate to a great extent) validity in psychometrics. Classical techniques include multi-trait multi-method (MTMM) analysis (Campbell & Fiske, 1959). An illustration of how it works is shown below (three methods or instruments, three constructs or traits):

enter image description here

In this MTMM matrix, the diagonal elements might be Cronbach's alpha or test-retest intraclass correlation; these are indicators of the reliability of each measurement scale. The validity of the hypothesized (shared) constructs is assessed by the correlation of scales scores when different instruments are used to assess the same trait; if these instrument were developed independently, high correlation ($> 0.7$) would support the idea that the traits are defined in a consistent and objective manner. The remaining cells in this MTMM matrix summarize relations between traits within method, and between traits across methods, and are indicative of the way unique constructs are measured with different scales and what are the relations between each trait in a given scale. Assuming independent traits, we generally don't expect them to be high (a recommended threshold is $<.3$), but more formal test of hypothesis (on correlation point estimates) can be carried out. A subtlety is that we use so-called "rest correlation", that is we compute correlation between an item (or trait) and its scale (or method) after removing the contribution of this item to the sum score of this scale (correction for overlap).

Even if this method was initially developed to assess convergent and discriminant validity of a certain number of traits as studied by different measurement instruments, it can be applied for a single multi-scale instrument. The traits then becomes the items, and the methods are just the different scales. A generalization of this method to a single instrument is also known as multitrait scaling. Items correlating as expected (i.e., with their own scale rather than a different scale) are counted as scaling success. We generally assume, however, that the different scales are not correlated, that is they are targeting different hypothetical constructs. But averaging the within and between-scale correlations provide a quick way of summarizing the internal structure of your instrument. Another convenient way of doing so is to apply a cluster analysis on the matrix of pairwise correlations and see how your variables do hang together.

Of note, in both cases, the usual caveats of working with correlation measures apply, that is you cannot account for measurement error, you need a large sample, instruments or tests are assumed to be "parallel" (tau-equivalence, uncorrelated errors, equal error variances).

The second part addressed by @rolando is also interesting: If there's no theoretical or substantive indication that the already established grouping of items makes sense, then you'll have to find a way to highlight the structure of your data with e.g., exploratory factor analysis. But even if you trust those "characteristics within a group", you can check that this is a valid assumption. Now, you might be using confirmatory factor analysis model to check that the pattern of items loadings (correlation of an item with its own scale) behaves as expected.

Instead of traditional factor analytic methods, you can also take a look at items clustering (Revelle, 1979) which relies on a Cronbach's alpha-based split-rule to group together items into homogeneous scales.

A final word: If you are using R, there are two very nice packages that will ease the aforementioned steps:

  • psych, provides you with everything you need for getting started with psychometrics methods, including factor analysis (fa, fa.parallel, principal), items clustering (ICLUST and related methods), Cronbach's alpha (alpha); there's a nice overview available on William Revelle's website, especially An introduction to psychometric theory with applications in R.
  • psy, also includes scree plot (via PCA + simulated datasets) visualization (scree.plot) and MTMM (mtmm).


  1. Campbell, D.T. and Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56: 81–105.
  2. Hays, R.D. and Fayers, P. (2005). Evaluating multi-item scales. In Assessing quality of life in clinical trials, (Fayers, P. and Hays, R., Eds.), pp. 41-53. Oxford.
  3. Revelle, W. (1979). Hierarchical Cluster Analysis and the Internal Structure of Tests. Multivariate Behavioral Research, 14: 57-74.
  • 1
    $\begingroup$ This is, perhaps, the most interesting response I've read on any of the Exchanges and I've been studying econometrics for 5 years. $\endgroup$
    – d8aninja
    Apr 14, 2016 at 13:42
  • $\begingroup$ Is the MTMM matrix included here supposed to be a realistic example of a correlation matrix? If so, I'll note that it's not actually a positive semidefinite matrix: for instance, the 4-by-4 minor for the correlations of traits 1,2 across methods 1,2 has determinant -0.0419179. (Since this is a small negative value, this perhaps simply goes to your point re: 'measurement error'.) $\endgroup$ Jun 11, 2018 at 14:13

The way I read your terminology, what you want is first to assess internal consistency within each group of variables, and then to assess the correlations among the scale scores which constitute the average of each group of variables. The first can be done using Cronbach's alpha, and the second using Pearson correlation. This assumes you have reasonably normal distributions and reasonably linear relationships.

A more involved method, and not necessarily a required one, would be to conduct an exploratory factor analysis. You would try to establish which variables should be grouped together and then again to what degree those factors would be correlated. If you try this method, make sure you use oblique rotation to allow those correlations to show up. Whether you use principal components extraction or principal axis extraction would depend, respectively, on whether your variables are objective, error-free measurements or subjective ones such as survey items that contain a certain amount of error.

  • $\begingroup$ Thanks for your response. I've managed to calculate Cronbach's alpha, but how does one calculate Pearson correlation coefficients in this case? I could calculate them in a pairwise fashion for each individual characteristic, but I'd like to know how to calculate correlations between groups of characteristics. Characteristics within a group will have similar scores for each observation. I'm going to edit my question to make this a little clearer. $\endgroup$
    – blep
    Apr 24, 2011 at 18:55
  • The standard tools, at least in psychology, in your situation would be exploratory and confirmatory factor analysis to assess the convergence of the inter-item correlation matrix with some proposed model of the relationship between factors and items. The way that you have phrased your question suggests that you might not be familiar with this literature. For example, here are my notes on the scale construction and factor analysis and here is a tutorial in R on factor analysis form Quick-R. Thus, while it's worth answering your specific question, I think that your broader aims will be better served by examining factor analytic approaches to evaluating multi-item, multi-factor scales.

  • Another standard strategy would be to calculate total scores for each group of variables (what I would call a "scale") and correlate the scales.

  • Many reliability analysis tools will report average inter-item correlation.

  • If you created the 50 by 50 matrix of correlations between items, you could write a function in R that averaged subsets based on combinations of groups of variables. You might not get what you want if you have a mixture of positive and negative items, as the negative correlations might cancel out the positive correlations.


I would suggest using as a replacement for the notion of correlation, which is defined only for pair-wise, the notion of mutual information and integration in Gaussian models.

In Gaussian models, integration of a group of variables $G_1$ is defined as the entropy of the group:

$I_1 \propto log(|C_1|)$

where $C_1$ is the correlation matrix of the group of variables $G_1$. It is easy to see that if $G_1$ is comprised only of 2 variables, its integration is $log ( 1 - \rho^2)$, which directly relates to the pairwise correlation coefficient of the variables $\rho$.

To compute interaction between two groups of variables, you can use mutual information, which is just cross-entropy between the groups:

$MU_{12} = I_{12} - I_{1} - I_{2}$

I found a reference on these notions after a quick google that might be helpful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.