5
$\begingroup$

I'm looking for a measure of dispersion, such as standard deviation, that can be used when distributing to an unordered set.

Specifically: A bucket distribution assigns a non-negative value to each bucket in a finite set. The sum of all assigned values is one. (So far, it's like a probability distribution). However: The buckets in the set can be distinguished but have no order. It's this which makes it different than a probability distribution.

We can represent bucket distributions as lists of decreasing non-negative numbers that sum to 1. Eg [1] or [1/2,1/2] or [1/3,1/3,1/6,1/6] or [1/2,1/4,1/8,1/16...].

I'd like to be able to measure how dispersed a particular bucket distribution is. Intuitively, [1] has 0 dispersion, [9/10,1/10] has some, [9/10,1/20,1/20] has more, [1/3,1/3,1/6,1/6] has more, etc. But I haven't been able to quantify this.

I've tried using standard deviation, variance, moment of inertia, etc. But I can't find a good way to do this. I can arbitrarily rank order the buckets, but this seems, well, arbitrary.

(I'm the original poster, but can't seem to comment any more)

Ray: Great work, thanks for sharing this original research. If you can provide more info on how you developed it, it would be fascinating.

whuber: Do you still feel entropy is a better measure than $k'$? Why? Intuitively, $k'$ fits like a glove.

Just noticed: The Wikipedia entry http://en.wikipedia.org/wiki/Diversity_index gives both $k'$, entropy, and other measures of diversity. But, alas, no comparison (what are fundamental assumptions of each, when is one better suited, etc.)

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Look up "entropy." $\endgroup$
    – whuber ♦
    Commented Feb 12, 2013 at 17:46
  • $\begingroup$ @whuber, I'm familiar with entropy: sum of (prob * log of prob). Can you elaborate a bit on how to apply it here? $\endgroup$
    – S. Robert James
    Commented Feb 12, 2013 at 17:49
  • $\begingroup$ You just gave the formula! You do have a probability measure (that's what you mean by "bucket distribution"); what you don't have--and it's irrelevant to entropy anyway--is a random variable. $\endgroup$
    – whuber ♦
    Commented Feb 12, 2013 at 17:51
  • $\begingroup$ @whuber Wow, what you're writing seems very interesting - really making me want to know more. But I'm afraid I'm unable to understand your comment entirely. I understand that I have a prob. measure. I don't think I have a prob. distribution. Hmmm... your comments are intriguing, if you'd elaborate more I'd be grateful. $\endgroup$
    – S. Robert James
    Commented Feb 12, 2013 at 18:01
  • $\begingroup$ Based on a search, I suggest you start with stats.stackexchange.com/questions/10789 and stats.stackexchange.com/questions/30999. A random variable is a real-valued function $X$ on your "buckets" (which are more often modeled as tickets in a box or balls in an urn). The distribution of $X$ is given by the function $F_X(x)$ = $\Pr(X \le x)$, which is computed by adding the probabilities of all "buckets" where $X$ has a value of $x$ or less. $\endgroup$
    – whuber ♦
    Commented Feb 12, 2013 at 18:10

1 Answer 1

Reset to default
2
$\begingroup$

$k' = 1 / \sum{p^2}$ is the "effective" number of buckets over which the distribution is uniform. $1 \le k' \le k =$ the # of buckets.

Edit

  1. Thanks for adding the markup. Also, I'm new to this site, and I don't know how to reply to a comment, as opposed to replying to a post.

  2. I developed k' myself, sometime in the 1970's, in response to a student's request for a way of measuring the variability across categories of admission diagnoses to a mental hospital, the question being whether there was more variability at certain times than at others. Recently, I discovered that it had previously been proposed by E.H. Simpson in a short article: Measurement of Diversity, Nature, 163 (1949), 688.

k' is not restricted to uniform distributions. Calling it the effective number of categories over which the distribution is uniformly distributed is just a way of establishing the scale it's on. It's analogous to the df in repeated measures anova after correcting for non-sphericity, with the normalized eigenvalues of the covariance matrix replacing the p's.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thanks Ray. (I added the markup to your post, but someone needs to approve it...) I see how if the distribution is uniform, $k'$ is the number of buckets. But can you justify why $k'$ makes sense when the distribution isn't uniform? $\endgroup$
    – S. Robert James
    Commented Feb 12, 2013 at 19:20
  • 2
    $\begingroup$ Also: Where did you get the idea from? Is this formula used elsewhere? $\endgroup$
    – S. Robert James
    Commented Feb 12, 2013 at 19:25
  • $\begingroup$ +1 Thank you for responding to the comments Ray, and welcome to our site! I have incorporated your separate answer as edits to this one. In general, consider editing your work when you wish to improve it and use the associated comments for material that might not properly be part of your answer. This provides a relatively clean separation between your work product and the commentary that may be associated with its development. The separation is important because the site treats answers differently than comments (in terms of searchability and so on). $\endgroup$
    – whuber ♦
    Commented Feb 12, 2013 at 23:06
  • $\begingroup$ This is also the inverse of the Herfindahl-Hirschman Index, which is a measure of competitiveness of an industry. $\endgroup$
    – dimitriy
    Commented Feb 12, 2013 at 23:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.