Should I use chi-squared?

Question

I have a dataset reporting a number of incidents for a given population, such as:

Country Nb incidents    Population  Rate
A             1               30    3.33%
B            71            15000    0.47%
C             4             2000    0.20%
D             1              600    0.17%
E            19            12000    0.16%
F             3                3     100%

My boss asked me to ignore countries having only one incident. Country A is incorrectly flagged as a problem, so we should not take it into account at all and concentrate on country B instead.

But I'd like to implement a more robust and efficient solution. My problem is that country A has the highest Rate of incidents, but it is obviously not significant since the tested population is very small. Country D has also only one incident but the tested population is large enough.

As an extreme example, I have added a country F where the rate is 5000%. Here, we have to consider the result because a lot of incidents for a small population would mean we have a huge problem ;)

I thought chi-squared would be OK, but I have no idea how I could use it on this data set.

Answer 1

I would not use the chi-squared test for this. You don't need a test to see if the proportions differ, you believe they differ—that's why you are trying to determine the 'worst'. There can't be a worst if they are all equal. Instead, you want a ranking. To get a ranking, we need to estimate some underlying quantity on which the ranking is based. In your case, you need to estimate an underlying probability (I think, or maybe the rate) of having an incident.

The problem is that you have differing amounts of information on which to base your estimates. This is actually a pretty common problem in a ranking context. What you need is something that will adjust the observed proportion itself to account for the differences in information. I suggest a more or less Bayesian approach. First, I suspect the different types of incidents are correlated due to common issues like better or worse maintenance (or perhaps proximity to the sea). I would compile a matrix from all countries (in rows) and all incident types (in columns), with each cell being the observed proportion. Then, I would compute the correlation matrix by columns and run a principle components analysis. See how many PCs seem reasonable. My guess is that you can cluster your incident types into at most a few groups, possibly even only one. Within each group of incidents, average the observed proportion within each incident type, and then average over the averages. (If you have a complete, balanced dataset, this is the average over the matrix.) This becomes your baseline proportion for all incidents within that group. Then determine the population sizes, and compute their average likewise. This is how much information the baseline average is typically based on. (At one extreme, you would have a single baseline and population average for the entire dataset, at the other extreme, you would have a different baseline and population average for each incident type.) With these two pieces of information, you can adjust your observed proportions. Specifically, just do a weighted average of your observed proportion with your baseline proportion, with the observed population size and average population size as the weights. All estimated proportions will be moved at least somewhat towards the baseline average, but small populations will be moved more and large populations will be moved less. That's it. The worst country is the one with the worst adjusted probability of incidents.