Rank forecasting from uneven samples

Say we have a bag of chocolate balls. There are $I$ different unique colors. Our bag has an unknown number of balls for each color.

We want to get a sense of which color people like the most and rank them accordingly.

We find $P$ people who volunteered to test our chocolate balls. Some of them are hungrier than others at the time of testing.

We proceed to collect data as follows:

1. For each person, we first ask them how many chocolate balls they want to eat.
2. We take a random sample with size matching the number of chocolate balls they want from the bag, and once they eat them, we finally ask them to rank the colors they tested.

What would be a good estimator of the ranking of colors?

Would the sample mean of the percentiles help us build a reasonable estimator of the $E[\text{ranking}]$ of colors in the population? Can we do better?

Forecasting

Assuming that new individuals will draw samples from a similar bag of chocolates (i.e. with the same unknown distribution of colors), our goal is to minimize the expected error in our percentile prediction, per color, assuming that the particular color is tested.

• Something to be aware of is that a ranking at the population level may not exist. Imagine Person 1 prefers Red to Blue to Green. Person 2 prefers Blue to Green to Red. Person 3 prefers Green to Red to Blue. Then 2/3 prefer Red to Blue, 2/3 prefer Blue to Green, and 2/3 prefer Green to Red. The preferences expressed by voting in this case violate the transitive property. – Matthew Gunn Mar 28 '17 at 22:18
• I share the concerns voiced by @Matthew: could you explain what you mean by "the ranking ... in the population"? Without some additional assumptions, this would seem not to have a definite meaning. Why should we suppose there is some universal ranking that applies to every individual? – whuber Mar 28 '17 at 23:01
• If you just want to know the most preferred color in your population (rather then the ranking) then I think your estimator will be simply the one that received the highest vote (this looks similar to some Max Likelihood estimation problem where the range of the likelihood changes with your sample, think of estimating the parameter of a uniform distribution) – Three Diag Mar 29 '17 at 10:51
• @whuber Thanks, it may help if I reframe this with the goal of minimizing error in a forecast of the ranking of balls for a generic person. That is, different people may draw (eat) a different number of balls, but we can still ask ourselves: what's the best we could do at (1) predicting what balls a person would eat, and (2) what ranking the person may establish. Clearly one could forecast one ranking for each possible combination of balls. Assuming that we have that, the forecast for the specific combination where all ball types are selected may give us "a full ranking" as well. – Amelio Vazquez-Reina Feb 23 '18 at 21:06