Skip to main content
added 81 characters in body
Source Link
Tim
  • 141.2k
  • 26
  • 270
  • 512

FromYou can always look at the variance of the counts, but looking at your description, entropy seems to be a measure that fitsnatural choice, since it meets all of your criteria. Entropy is defined as

$$ S = -\sum_i p_i \log p_i $$

where $p_i$ is a probability of observing $i$-th category. The more uniform would be the distribution, the higher entropy it displays, so it is about being "diverse" vs uniform.

In your case, you have counts, so you can use them to calculate the empirical probabilities

$$ \hat p_i = \frac{n_i}{\sum_j n_j} $$

where $n_i$ is the count for the $i$-th category. Since you have exact zeros in the counts, you should use some estimator of the probabilities that "smoothes" the zeros, since otherwise the formula for the entropy would not work (single zero would zero-out everything), one approach could be using a Bayesian estimator like Laplace smoothing, i.e.

$$ \hat p_i = \frac{n_i+\alpha}{\sum_j n_j+\alpha} $$

where $\alpha$ is some constant, e.g. $\alpha=1$. In R this translates to:

> prob <- function(n, alpha=1) (n+alpha)/sum(n+alpha)
> entropy <- function(p) -sum(p*log(p))
> entropy(prob(c(0, 0, 100)))
[1] 0.1092225
> entropy(prob(c(20, 20, 20)))
[1] 1.098612
> entropy(prob(c(10, 0, 10)))
[1] 0.8418553
> entropy(prob(c(0, 10, 10)))
[1] 0.8418553

As you can see, sample 1 has low entropy, while sample 2 has high entropy. For samples 3 and 4, the entropy is the same, higher then for sample 1 (they are less extreme), but lower then sample 2 that is uniformly distributed.

From your description, entropy seems to be a measure that fits your criteria. Entropy is defined as

$$ S = -\sum_i p_i \log p_i $$

where $p_i$ is a probability of observing $i$-th category. The more uniform would be the distribution, the higher entropy it displays, so it is about being "diverse" vs uniform.

In your case, you have counts, so you can use them to calculate the empirical probabilities

$$ \hat p_i = \frac{n_i}{\sum_j n_j} $$

where $n_i$ is the count for the $i$-th category. Since you have exact zeros in the counts, you should use some estimator of the probabilities that "smoothes" the zeros, since otherwise the formula for the entropy would not work (single zero would zero-out everything), one approach could be using a Bayesian estimator like Laplace smoothing, i.e.

$$ \hat p_i = \frac{n_i+\alpha}{\sum_j n_j+\alpha} $$

where $\alpha$ is some constant, e.g. $\alpha=1$. In R this translates to:

> prob <- function(n, alpha=1) (n+alpha)/sum(n+alpha)
> entropy <- function(p) -sum(p*log(p))
> entropy(prob(c(0, 0, 100)))
[1] 0.1092225
> entropy(prob(c(20, 20, 20)))
[1] 1.098612
> entropy(prob(c(10, 0, 10)))
[1] 0.8418553
> entropy(prob(c(0, 10, 10)))
[1] 0.8418553

As you can see, sample 1 has low entropy, while sample 2 has high entropy. For samples 3 and 4, the entropy is the same, higher then for sample 1 (they are less extreme), but lower then sample 2 that is uniformly distributed.

You can always look at the variance of the counts, but looking at your description, entropy seems to be a natural choice, since it meets all of your criteria. Entropy is defined as

$$ S = -\sum_i p_i \log p_i $$

where $p_i$ is a probability of observing $i$-th category. The more uniform would be the distribution, the higher entropy it displays, so it is about being "diverse" vs uniform.

In your case, you have counts, so you can use them to calculate the empirical probabilities

$$ \hat p_i = \frac{n_i}{\sum_j n_j} $$

where $n_i$ is the count for the $i$-th category. Since you have exact zeros in the counts, you should use some estimator of the probabilities that "smoothes" the zeros, since otherwise the formula for the entropy would not work (single zero would zero-out everything), one approach could be using a Bayesian estimator like Laplace smoothing, i.e.

$$ \hat p_i = \frac{n_i+\alpha}{\sum_j n_j+\alpha} $$

where $\alpha$ is some constant, e.g. $\alpha=1$. In R this translates to:

> prob <- function(n, alpha=1) (n+alpha)/sum(n+alpha)
> entropy <- function(p) -sum(p*log(p))
> entropy(prob(c(0, 0, 100)))
[1] 0.1092225
> entropy(prob(c(20, 20, 20)))
[1] 1.098612
> entropy(prob(c(10, 0, 10)))
[1] 0.8418553
> entropy(prob(c(0, 10, 10)))
[1] 0.8418553

As you can see, sample 1 has low entropy, while sample 2 has high entropy. For samples 3 and 4, the entropy is the same, higher then for sample 1 (they are less extreme), but lower then sample 2 that is uniformly distributed.

Source Link
Tim
  • 141.2k
  • 26
  • 270
  • 512

From your description, entropy seems to be a measure that fits your criteria. Entropy is defined as

$$ S = -\sum_i p_i \log p_i $$

where $p_i$ is a probability of observing $i$-th category. The more uniform would be the distribution, the higher entropy it displays, so it is about being "diverse" vs uniform.

In your case, you have counts, so you can use them to calculate the empirical probabilities

$$ \hat p_i = \frac{n_i}{\sum_j n_j} $$

where $n_i$ is the count for the $i$-th category. Since you have exact zeros in the counts, you should use some estimator of the probabilities that "smoothes" the zeros, since otherwise the formula for the entropy would not work (single zero would zero-out everything), one approach could be using a Bayesian estimator like Laplace smoothing, i.e.

$$ \hat p_i = \frac{n_i+\alpha}{\sum_j n_j+\alpha} $$

where $\alpha$ is some constant, e.g. $\alpha=1$. In R this translates to:

> prob <- function(n, alpha=1) (n+alpha)/sum(n+alpha)
> entropy <- function(p) -sum(p*log(p))
> entropy(prob(c(0, 0, 100)))
[1] 0.1092225
> entropy(prob(c(20, 20, 20)))
[1] 1.098612
> entropy(prob(c(10, 0, 10)))
[1] 0.8418553
> entropy(prob(c(0, 10, 10)))
[1] 0.8418553

As you can see, sample 1 has low entropy, while sample 2 has high entropy. For samples 3 and 4, the entropy is the same, higher then for sample 1 (they are less extreme), but lower then sample 2 that is uniformly distributed.