Quantifying homogeneity

Question

I am hydrologist and I am looking for a solution for the following problem:

Suppose that I have $k$ regions each yielding a sample of the same variable (e.g., annual peak flow) drawn from different sites within the given region (each sample $i=1,..K$ has a record length $n_i$).

The homogeneity test (e.g., $K$ sample Anderson-Darling test) answers the question: is the region homogeneous? As a null hypothesis test, there are only two possibilities of answer: i.e., either the region is homogeneous ($F_1=F_2=...=F_k=F$) or it is heterogeneous. This binary answer is not flexible and do not allow me to compare homogeneity among several regions to point out the most homogeneous of them.

So I am looking for a measure that overcomes this limitation. By analogy, this measure would be the same as the Akaike Information Criterion (AIC) for linear models. Indeed, the AIC compares different models in order to determine which one is the better to fit to data. Thus, this measure would compare different region and measure of how much sampling sites are alike within regions.

Answer 1

The Anderson Darling test is a test of whether samples come from the same distribution. (See, e.g. [Wikipedia http://en.wikipedia.org/wiki/Anderson%E2%80%93Darling_test]. That must have a binary answer.

I am not completely clear on what data you have. If you have $n_k$ data points from each of $k$ regions, you can see the variation (heterogeneity) in each region by using the variance or standard deviation or interquartile range or any of a number of other measures of spread.

Answer 2

If the sample-sizes are similar, you could compare Anderson-Darling statistics pairwise to see which ones are more alike and which ones more different from others.

More likely to be relevant still would be some kind of classification or clustering algorithm; indeed, such an algorithm could even be based off goodness-of-fit measures like a K-S or A-D which might be used to build a dissimilarity matrix (indeed, K-S is quite explicitly a distance).

Thus, this measure would compare different region and measure of how much sampling sites are alike within regions.

It's not particularly clear how you envision this working. Can you clarify?

Answer 3

Presumably the within region standard deviation or variance would quantify the heterogeneity within regions.

You could quantify the variability in region variances in an informal way by taking the standard deviation of the within-region standard deviations.

A better approach might be to fit a multilevel model where you allow the size of within region variances to vary across regions. You could look at different models of this variability in within-region variances. This could include models where there is no such variance, to various priors on the variability. Note I've never done this before myself so I'm not quite sure what identifiability issues you might run into.