How to determine the "health" of a population, considering its size

Question

I am interested in the unhealthy people across 10 different populations. Here is the raw data:

Popn   Unhealthy    Size   Percentage
----  ---------- ------- ------------
A            170    1000        17.0%
B              2       2       100.0%
C             12      25        52.0%
D              7      20        35.0%
E              4      13        30.8%
F             13      54        24.1%
G             11      23        47.8%
H             14      42        33.3%
I              1       3        33.3%
J              9      32        28.1%
      ---------- -------
             244    1214

                       Average: 40.1%
            Standard Deviation: 23.4%

My problem is I do not know how to best sort this data without being skewed by the outliers at A and B. If I sort the rows by the number of unhealthy people then A comes out on top because it has way more people in it, and always provides the largest number, but its percentage is a quite healthy looking 17%.

If I sort by percentage of healthy people per population size then B comes out on top because it has 100%; however B is only made up of two people. It doesn't strike me that two people is statistically relevant enough to warrant more concern than, say, the 52% of unhealthy people in population C.

Is there something I can calculate for each population which will help?

Answer 1

Here's a fairly ad-hoc answer. Let's call an unhealthy person a 'success' (bear with me). You want to know what fraction of the population are successes, given the observed data.

If you observe that there are $k$ successes in $n$ trials, then that might imply that a total fraction $f=k/n$ of the population are successes. However, this is clearly nonsensical if we only observe one member of the population. Instead, Laplace's rule of succession tells us that a better estimate for the probability of drawing a success from the population is

$$p = \frac{k+1}{n+2}$$

If this was the true probability, then the process generating your data is binomial with parameters $n$ and $p$. The mean of such a process is $np$ and the variance (square of the standard deviation) is $np(1-p)$. In terms of your observed data, the mean number of successes is

$$\frac{n(k+1)}{n+2} = \frac{n(f+1/n)}{1 + 2/n}$$

(note that this tends to $nf$ as $n\to\infty$) and the variance is

$$\frac{n(k+1)(n+1-k)}{(n+2)^2}$$

Now, your observed mean is itself only an estimate, and it has its own standard deviation. The standard deviation of the sample mean is known to decrease with the square root of the number of samples, so the standard deviation of the sample mean is

$$\sqrt{\frac{(k+1)(n+1-k)}{(n+2)^2}} = \sqrt{\frac{(f+1/n)(1-f + 1/n)}{(1 + 2/n)^2}}$$

which tends to $f(1-f)$ as $n\to\infty$. As a rough approximation, we can be 95% sure that our observed mean is no more than two standard deviations away from the actual population mean, which gives us a formula for the minimum value of the actual mean (with 95% certainty):

$$\frac{n(f+1/n) - 2\sqrt{(f+1/n)(1-f + 1/n)}}{(1 + 2/n)}$$

For example, with 2 observations, both of which are unhealthy, we estimate that the minimum value for the mean number of unhealthy people we observed would be 0.633, rather than 2. Dividing by the sample size gives us that at least 31.7% of the population are unhealthy.

On the other hand, with 25 samples of which 12 are unhealthy, we would expect to see at least 11.0 healthy people in our sample, which gives a minimum proportion of unhealthy people of 44.2%, which is much higher than 31.7%.

So by estimating the error in our measurements and taking the lowest estimate for the fraction of unhealthy people consistent with our data, we see that population C is much more at risk than population B, even though a smaller fraction of the observed population was unhealthy.

---

I wrote a quick script in R to calculate these values for your data, here are the results:

     k    n      min_p
A  170 1000 0.16990626
B    2    2 0.31698730
C   12   25 0.44150893
D    7   20 0.31553179
E    4   13 0.26080956
F   13   54 0.23396249
G   11   23 0.43655654
H   14   42 0.31833695
I    1    3 0.07340137
J    9   32 0.26563983

It's clear that the most at-risk population by this metric are groups C and G.

Answer 2

If you attempt to sort on a single value, you are implicitly making a trade-off between the estimated proportion and the uncertainty. In many cases there is no good basis for such a trade-off. Moreover, the uncertainty in the means attaches to the uncertainties in any upper or lower bounds you might place on the means, so using bounds alone as the basis of sorting solves nothing. (E.g., the $(k+1)/(n+2)$ estimator, although it is a valid Bayes estimate, has an artificial ring to it. It's equivalent to assuming your "prior belief" is the same as if you had earlier observed one healthy and one unhealthy person in each group. And the basis for that assumption is?)

What to do? If the purpose is to understand or present the data, consider plotting them. On the plot you can show the uncertainty estimates as "error bars." For example, the following figure shows two-sided 95% Poisson confidence limits around the estimates, sorted by descending value.

(For these purposes any reasonable way to compute confidence limits should be fine.) Graphing is more powerful than a simple tabulation or sort of the data because it simultaneously shows the estimates and the ranges of values that are plausibly consistent with them. Thus, for instance, the mean for group A clearly is less than the means for groups C, G, D, and H, but is not necessarily less than the means for groups B, I, E, J, or F. It now becomes evident, even to the casual onlooker, that these data do not clearly distinguish among most of the groups. Indeed, a chi-squared test for groups B-J (which are distinguished from group A by being much smaller in size) does not indicate any difference at all among their means (p = 19.3%, df=8). Merely sorting the numbers would not be likely to suggest this possibility.