Statistical method to quantify  diversity / variance / inequality I am looking for a statistical method to define the variance/ diversity / inequality in a set of observations. 
For example: 
If I have following (n=4) observations using 4 data points, here the diversity/variance/inequality is zero. 
A-B-C-D
A-B-C-D
A-B-C-D
A-B-C-D 

In the following  (n=4) observations using 7 data points, there is x amount of diversity / variance / inequality in the observations. What method I can use to derive a diversity / variance score ?  
A-B-B-D
Q-D-C-B
B-C-B-A
B-Z-F-A

My real data is derived from a database of 10K data points and observations are usually >= 2 - 1000s. Currently am looking at Gini Impurity as a potential method, Do you think Gini Impurity is better for this type data ? Do you know about any better method which can derive a score by considering the order of the data ? Looking forward for your suggestions. 
 A: I can't give a specific answer - mainly because the question isn't specific enough (happy to edit in due course though, given more information).  If the values A, B, C, D, E, etc. have a definite ordering that is meaningful to you (such as $A>C>B>E>Z>\dots$) and you only want to describe the diversity that exists within your observations, then gini coefficient makes sense.
However, if the values A, B, C, etc. are just arbitrary labels (i.e. they carry no information apart from distinguishing one type from another type) then a gini coefficient makes less sense - what is the relevant analogy to "rich" and "poor" if being "rich" can't be seen to be greater than being "poor".  For Gini coefficient to work, you have to be able to order the observations
A further consideration is if you wish to make inference about something which exists outside your sampled values.  For example, if the first set of 4 observations (which are the same) are part of a larger group which was not observed and you want to make a statement about the larger group.  Such a statement might be the sample is not diverse therefore the group that the sample came from is not diverse.  Gini co-efficient may or may not be appropriate in this case, depending on what you know about how the larger group is related to the sample, and on how big your sample is compared to the larger group.
One way to think about the problem which may help is to consider the following scenarios.  If you were told that a set of observations A is diverse, but not given any observations from A, what would you predict them to be?  What if you were given the first observation only?  If you were told that the set A is more diverse than another set B, and you were given the observations from set B, what would you predict the observations in set A to be?  Thinking about the problem this way will help you to describe the features that your diversity measure should have (and features that it shouldn't have).
If your data are categorical, then you could use tests based on a multinomial distribution with the number of trials equal to the number of categories per observation (4 in your examples) and the number of probability parameters equal to the number of data points (4 in your first example, and 7 in your second).  So, taking the second example we have:
$$x_{i}\sim Multinomial(4,\theta_{1},\theta_{2},\dots,\theta_{7})$$
And a homogeneity score can be created by calculating the probability that all of the $\theta_{j}$ are equal.  However, in calculating this probability (which is not difficult) you will also end up calculate the probability of several other hypothesis (which are also useful), such as $\theta_{1}$ is different, all other $\theta_{j}$ are equal, that 2 are different, 3 are different, and so on up to all 7 are different.  I can post how you would do this, but I want to make sure this is something that you actually want first!  If you also cared about positioning (so that $A-B-C-D$ is considered different to $A-C-B-D$), then this can be incorporated by creating $28$ $\theta$ pararemters, and doing hypothesis tests about them.  So you would have:
$$x_{ik}\sim Multinomial(1,\theta_{1k},\theta_{2k},\dots,\theta_{7k})\;\;\;\;\;\;\;\;k=1,2,3,4$$
Admittedly you will need a reasonably sized data set in order to do this kind of test.  And you would have hypothesis about the $k$ index indicating "ordering" diversity, and hypothesis about the $j$ index indicating "composition" diversity.
