Is there a ranking metric based on percentages that favors larger magnitudes?

Question

I have two groups, "in" and "out," and item categories that can be split up among the groups. For example, I can have item category A that is 99% "in" and 1% "out," and item B that is 98% "in" and 2% "out."

For each of these items, I actually have the counts that are in/out. For example, A could have 99 items in and 1 item out, and B could have 196 items that are in and 4 that are out.

I would like to rank these items based on the percentage that are "in," but I would also like to give some priority to items that have larger overall populations. This is because I would like to focus on items that are very relevant to the "in" group, but still have a large number of items in the "out" group that I could pursue.

Is there some kind of score that could do this?

edit: I should add that I also know the total number of items that are in and out across all categories. I have a pretty complete picture of how many items I have, what categories they are in, and whether or not they are in/out.

Answer 1

You can simply calculate proportion of 'in',

$$ P = {n_{in} \over n}, $$

but - as per your question - it's better to add a correction for total number of 'votes'.

One way to correct is to add some dummy 'outs' (e.g. $10$), so

$$ P' = {n_{in} \over n + 10 }. $$ items with a large number of votes see their modified percentage alters very little from their real percentage, but items with relatively few answers will see their modified percentage move considerably toward low values.

This is known as "Bayesian averaging". In effect, the items with many votes will rank higher than items with the same percentage but fewer votes.

In your example $P'_A=0.9$ and $P'_B=0.93$.

Answer 2

I don't know if there is a "standard" way to do this, but one approach is to scale the percent of each category by the number of "in" or the size of the category. For example, you could multiply the percent in by:

• size of category (equivalent to counting the number of "in")
• sqrt(size of category)
• ln(size of cateogry)

You would need to select a function (one of the above or some other) that is most appropriate for your application.