I am looking for a statistical method to define the variance/ diversity / inequality in a set of observations.

For example: If I have following (n=4) observations using 4 data points, here the diversity/variance/inequality is zero.


In the following (n=4) observations using 7 data points, there is x amount of diversity / variance / inequality in the observations. What method I can use to derive a diversity / variance score ?


My real data is derived from a database of 10K data points and observations are usually >= 2 - 1000s. Currently am looking at Gini Impurity as a potential method, Do you think Gini Impurity is better for this type data ? Do you know about any better method which can derive a score by considering the order of the data ? Looking forward for your suggestions.

  • 2
    $\begingroup$ You should at least mention whether order is a part of your 'similarity' score. $\endgroup$ – Nick Sabbe May 17 '11 at 22:02
  • 1
    $\begingroup$ @Shameer I was curious about a few things in your problem: Are data points the same as categorical attributes? What does your observational unit represent in the real-world? What are the 10K data points actually representing in the real-world. Is it human and human attributes? When you refer to inequality do you refer to the concern regarding human inequality or simply dispersion? Should each 'data point' be assumed to hold the same weight? Thanks... $\endgroup$ – b_dev May 18 '11 at 0:51
  • $\begingroup$ @Nick, I need to consider the order as part of my score. Thanks for this, now I have I edited the question. $\endgroup$ – Khader Shameer May 18 '11 at 21:38
  • $\begingroup$ @B_Dev, The data is kind of a categorical label. I used A, B, C, D... to simplify the representation. In real-world these are protein domains(see: en.wikipedia.org/wiki/Protein_domain) encoded in a protein sequence sourced from a database of protein domains (for example see: pfam.sanger.ac.uk). Here, 1 row = 1 protein with different domains. I am looking at dispersion, but am looking for a sequence-alignment free method here. I can assign weights to data points, but that will be an extension of my analysis. $\endgroup$ – Khader Shameer May 18 '11 at 21:51

I can't give a specific answer - mainly because the question isn't specific enough (happy to edit in due course though, given more information). If the values A, B, C, D, E, etc. have a definite ordering that is meaningful to you (such as $A>C>B>E>Z>\dots$) and you only want to describe the diversity that exists within your observations, then gini coefficient makes sense.

However, if the values A, B, C, etc. are just arbitrary labels (i.e. they carry no information apart from distinguishing one type from another type) then a gini coefficient makes less sense - what is the relevant analogy to "rich" and "poor" if being "rich" can't be seen to be greater than being "poor". For Gini coefficient to work, you have to be able to order the observations

A further consideration is if you wish to make inference about something which exists outside your sampled values. For example, if the first set of 4 observations (which are the same) are part of a larger group which was not observed and you want to make a statement about the larger group. Such a statement might be the sample is not diverse therefore the group that the sample came from is not diverse. Gini co-efficient may or may not be appropriate in this case, depending on what you know about how the larger group is related to the sample, and on how big your sample is compared to the larger group.

One way to think about the problem which may help is to consider the following scenarios. If you were told that a set of observations A is diverse, but not given any observations from A, what would you predict them to be? What if you were given the first observation only? If you were told that the set A is more diverse than another set B, and you were given the observations from set B, what would you predict the observations in set A to be? Thinking about the problem this way will help you to describe the features that your diversity measure should have (and features that it shouldn't have).

If your data are categorical, then you could use tests based on a multinomial distribution with the number of trials equal to the number of categories per observation (4 in your examples) and the number of probability parameters equal to the number of data points (4 in your first example, and 7 in your second). So, taking the second example we have:

$$x_{i}\sim Multinomial(4,\theta_{1},\theta_{2},\dots,\theta_{7})$$

And a homogeneity score can be created by calculating the probability that all of the $\theta_{j}$ are equal. However, in calculating this probability (which is not difficult) you will also end up calculate the probability of several other hypothesis (which are also useful), such as $\theta_{1}$ is different, all other $\theta_{j}$ are equal, that 2 are different, 3 are different, and so on up to all 7 are different. I can post how you would do this, but I want to make sure this is something that you actually want first! If you also cared about positioning (so that $A-B-C-D$ is considered different to $A-C-B-D$), then this can be incorporated by creating $28$ $\theta$ pararemters, and doing hypothesis tests about them. So you would have: $$x_{ik}\sim Multinomial(1,\theta_{1k},\theta_{2k},\dots,\theta_{7k})\;\;\;\;\;\;\;\;k=1,2,3,4$$ Admittedly you will need a reasonably sized data set in order to do this kind of test. And you would have hypothesis about the $k$ index indicating "ordering" diversity, and hypothesis about the $j$ index indicating "composition" diversity.

  • $\begingroup$ That's a wonderful answer ! Thanks a lot for your thoughts !!. I like the way you answered my concerns about applying Gini co-efficient to my problem. I will explore the possibilities of applying multinomial distribution here. $\endgroup$ – Khader Shameer May 18 '11 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.