# Measure heterogeneity among multiple samples and scale variables

I would like to know what are the available statistical measures for quantifying heterogeneity on data that look like this...

Sample1 Sample2 Sample3 Sample4
A 0.5 0.3 0.1 0.7
B 0.7 0.8 0.5 0.9
C 0.1 0.8 0.5 0.9
D 0.1 0.3 0.3 0.5
E 0.2 0.6 0.3 0.5

I've been using R^2 for two samples on some of my past research but now I need a tool that extends to more than 2 data sets. I am expecting to find a tool that takes into account the discrepancies among individual values, so that both of the examples below have essentially the same heterogeneity index (zero?)

Sample1 Sample2 Sample3 Sample4
A 0.5 0.5 0.5 0.5
B 0.5 0.5 0.5 0.5
C 0.5 0.5 0.5 0.5
D 0.5 0.5 0.5 0.5
E 0.5 0.5 0.5 0.5

Sample1 Sample2 Sample3 Sample4
A 0.1 0.1 0.1 0.1
B 0.1 0.1 0.1 0.1
C 0.1 0.1 0.1 0.1
D 0.1 0.1 0.1 0.1
E 0.1 0.1 0.1 0.1
• To get a useful answer you will have to be more specific about what you mean by "heterogeneity," what "A" through "E" represent, and what types of comparisons among Samples you anticipate. A bit more background on the nature of the underlying scientific problem might help. Also, if the data values are proportions then the answer might differ from a case where the values can in principle take on all real-number or all non-negative values.
– EdM
Apr 30, 2016 at 21:01
• The values are proportions indeed and A through E are basically different percentage variables that each sample is tested on. Academically speaking, the background is biomedical.
– civy
Apr 30, 2016 at 21:55
• That's a start, but you haven't specified whether you care about heterogeneity within variables among samples, or within samples among variables. It would also be help to know more about the raw data on which the proportions are based: for example, are they successes among trials and if so how many trials? Note that useful estimates of heterogeneity require large numbers of cases.
– EdM
May 1, 2016 at 0:58
• I guess that I want both (variables among samples and samples among variables). The fractions are calculated based on 1000 to 1 million counts so there's a considerable number of cases to support reliability if that's what you mean.
– civy
May 1, 2016 at 6:21
• I'm still a bit unclear on what you mean by "heterogeneity" and how you used $R^2$ values to measure it. If you mean "lack of correlation," I don't think that's a common use of the word "heterogeneity." Please say more about the hypotheses that you are trying to test.
– EdM
May 1, 2016 at 9:38

Your present measure of "heterogeneity" is presumably $1-R^2$ for the 2-sample case, so that if samples 1 and 2 agree for each category you have a result of 0, while if there is no correlation between the 2 samples with respect to category values (complete "heterogeneity") you have a result of 1.

The simplest way to extend this approach to multiple samples is to calculate the determinant of the correlation matrix. The correlation matrix displays the correlation coefficients for all pairs of the samples. The determinant of a covariance matrix can be used as a measure of overall dispersion; the correlation matrix removes the dependence of the covariance matrix on the scales of the variables, and restricts the values of the determinant to the interval [0,1]. For the 2-sample case this determinant is $1-R^2$, as you have been using. So for your first example (using R):

> mat1
sample1 sample2 sample3 sample4
A     0.5     0.8     0.5     0.5
B     0.3     0.5     0.9     0.2
C     0.1     0.9     0.1     0.6
D     0.7     0.1     0.3     0.3
E     0.7     0.8     0.3     0.5

the correlation matrix is:

> cor(mat1)
sample1    sample2     sample3    sample4
sample1  1.00000000 -0.4572127 -0.05057217 -0.2100420
sample2 -0.45721268  1.0000000 -0.23181527  0.7814038
sample3 -0.05057217 -0.2318153  1.00000000 -0.7624437
sample4 -0.21004201  0.7814038 -0.76244374  1.0000000

and its determinant is:

> det(cor(mat1))
[1] 0.02092142

You should, however, consider some of the limitations of this approach with data that represent proportions.

First, since proportions are limited to the range [0,1], measures of $R^2$ don't have the nice properties that you might expect from the usual presentations of correlations based on bivariate normal distributions. Depending on the nature of your data and how much you care about differences among proportions that are close to 0 or 1, you might consider a logit transformation of the proportions to spread them out over all real numbers (available directly as a function if you load the boot package in R):

$$\text{logit}(p)=\log\left(\frac{p}{1-p}\right),$$

although this will lead to problems if any of your proportions are exactly 0 or 1. Alternatively, you might consider a non-parametric correlation like Spearman's, available in R as cor(mat1,method="spearman"). That's basically the correlation coefficient calculated from the ranks of the values instead of from the values themselves.

Second, your should consider the danger in throwing away the information you have on the raw data underlying these proportions. If each proportion can be based on such a wide range of counts (1 thousand to 1 million) there might be some important relations of your samples or categories to the total number of counts that are important to understand, or the observed proportions might not be truly representative of the underlying phenomenon you think that you are measuring. A more complete statistical model that takes these possibilities into account could provide measures of "heterogeneity" that have a more solid basis.