# How does one measure the non-uniformity of a distribution?

I'm trying to come up with a metric for measuring non-uniformity of a distribution for an experiment I'm running. I have a random variable that should be uniformly distributed in most cases, and I'd like to be able to identify (and possibly measure the degree of) examples of data sets where the variable is not uniformly distributed within some margin.

An example of three data series each with 10 measurements representing frequency of the occurrence of something I'm measuring might be something like this:

a: [10% 11% 10%  9%  9% 11% 10% 10% 12%  8%]
b: [10% 10% 10%  8% 10% 10%  9%  9% 12%  8%]
c: [ 3%  2% 60%  2%  3%  7%  6%  5%  5%  7%]   <-- non-uniform
d: [98% 97% 99% 98% 98% 96% 99% 96% 99% 98%]


I'd like to be able to distinguish distributions like c from those like a and b, and measure c's deviation from a uniform distribution. Equivalently, if there's a metric for how uniform a distribution is (std. deviation close to zero?), I can perhaps use that to distinguish ones with high variance. However, my data may just have one or two outliers, like the c example above, and am not sure if that will be easily detectable that way.

I can hack something to do this in software, but am looking for statistical methods/approaches to justify this formally. I took a class years ago, but stats is not my area. This seems like something that should have a well-known approach. Sorry if any of this is completely bone-headed. Thanks in advance!

-

If you have not only the frequencies but the actual counts, you can use a $\chi^2$ goodness-of-fit test for each data series. In particular, you wish to use the test for a discrete uniform distribution. This gives you a good test, which allows you to find out which data series are likely not to have been generated by a uniform distribution, but does not provide a measure of uniformity.

There are other possible approaches, such as computing the entropy of each series - the uniform distribution maximizes the entropy, so if the entropy is suspiciously low you would conclude that you probably don't have a uniform distribution. That works as a measure of uniformity in some sense.

Another suggestion would be to use a measure like the Kullback-Leibler divergence, which measures the similarity of two distributions.

-
I have a couple of questions regarding your reply: 1. Why do you state that chi-square does not give a measure of uniformity? Isn't a fit test with a uniform distribution a measure of uniformity? 2. How can we know when should we use either chi-square or entropy ? – kanzen_master Nov 8 '12 at 5:44
@kanzen_master: I guess that the chi-squared statistic can be seen as a measure of uniformity, but it has some drawbacks, such as the lack of convergence, dependence on the arbitrarily placed bins, that the number of expected counts in the cells needs to be sufficiently large, etc. Which measure/test to use is a matter of taste though, and entropy is not without its problems either (in particular, there are many different estimators of the entropy of a distribution). To me, entropy seems like a less arbitrary measure and is easier to interpret. – MånsT Nov 19 '12 at 8:28

In addition to @MansT 's good ideas, you could come up with other measures, but it depends on what you mean by "non-uniformity". To keep it simple, let's look at 4 levels. Perfect uniformity is easy to define:

25 25 25 25

but which of the following is more non-uniform?

20 20 30 30 or 20 20 25 35

or are they equally non-uniform?

if you think they are equally non-uniform, you could use a measure based on the sum of the absolute values of the deviations from normal, scaled by the maximum possible. Then the first is 5 + 5 + 5 + 5 = 20 and the second is 5 + 5 + 0 + 10 = 20. But if you think the second is more nonuniform, you could use something based on the squared deviations in which case the first gets 25 + 25 + 25 + 25 = 100 and the second gets 25 + 25 + 0 + 100 = 150.

-
You seem to be interpreting "uniformly distributed" as "equal", Peter. Whether that is the OP's intention is a valid point to raise, but really should appear as a comment to the question. – whuber Apr 4 '12 at 12:52
Hi @whuber That seemed to be what he meant, from the question. What else might it mean? – Peter Flom Apr 5 '12 at 11:37
"Equal" means the CDF is $F(x) = 1$ for $x\ge \mu$, $F(x) = 0$ for $x\lt \mu$ while "uniform" means $F(x) = (x-\alpha)/\theta$ for $x \in [\alpha, \alpha+\theta]$. You define "perfect uniformity" in the first sense whereas the standard statistical sense is the second. – whuber Apr 5 '12 at 14:21
@whuber, it seems to me the first thing is closer to what the original poster meant by "uniform". Looking at it again, it seems like he/she was using "uniform" to mean "low variance". – Macro Apr 6 '12 at 12:40
That's just it, Macro: we cannot really say. The question needs clarification before it deserves an answer, IMHO. The accepted answer suggests the OP used "uniform" in the standard statistical sense. – whuber Apr 6 '12 at 15:52

Here is a simple heuristic: if you assume elements in any vector sum to $1$ (or simply normalize each element with the sum to achieve this), then uniformity can be represented by L2 norm, which ranges from $1/\sqrt d$ to $1$, with $d$ being the dimension of vectors. The lower bound $1/\sqrt d$ corresponds to uniformity and upper bound to the $1$-hot vector.

To scale this to a score between $0$ and $1$, you can use $(n*\sqrt d - 1)/(\sqrt d - 1)$, where $n$ is the L2 norm.

An example modified from urs --- with elements summing to $1$ and all vectors with the same dimension for simplicity:

0.10    0.11    0.10    0.09    0.09    0.11    0.10    0.10    0.12    0.08
0.10    0.10    0.10    0.08    0.12    0.12    0.09    0.09    0.12    0.08
0.03    0.02    0.61    0.02    0.03    0.07    0.06    0.05    0.06    0.05


The following will yield $0.0028$, $0.0051$, and $0.4529$ for the rows:

d=size(m,2);
for i=1:size(m);
disp( (norm(m(i,:))*sqrt(d)-1) / (sqrt(d)-1) );
end

-
That works nicely. But why (or under what circumstances) should it be preferred to any other $L_p$ norm or to the other solutions offered on this thread? – whuber Mar 31 '14 at 22:03
@whuber that I don't know and I don't know of any research on this. Basically it's something I've used as a heuristic that might fit what OP is after, and I don't really claim it being a preferred approach. – user495285 Apr 2 '14 at 22:42

Stumbled upon this recently, and to add to the answer from @user495285, as far as I understand it:

When the values are normalized and sum to one, then the uniform distribution is the unit sphere in $\mathbb{R}^n$, and what is being calculated by using an $L_p$ norm is the deviation from the unit sphere using a distance measure of a given $p$, i.e. deviation from the uniform distribution in $\mathbb{R}^n$ with geometric distance measure $p$.

The $L_2$ norm places higher weight on large deviations from the unit sphere in any given dimension, whereas smaller values of $p$ place less weight on large deviations.

When the underlying distribution is the unit sphere, the numerator equals zero in the following equation: $$\frac{n\sqrt{d} - 1}{\sqrt{d} - 1}$$ where $n$ is the $L_2$ norm and $d$ is the vector length.

I believe that the usefulness of geometric measures applies when each position (dimension) of the space described is assumed to be measured on equivalent scales, e.g. all counts of potentially equal distribution. The same assumptions underlying change of bases like PCA/SVD probably are similar here. But then again I'm no mathematician, so I'll leave that open to the more informed.

-