Survey data : calibrate result according to number of response I ll do my best to explain the situation.
I'm trying to rank X supermarkets according to their performance.
I've done a survey with a set of Y clients and obtained an average grade for each supermarket. 
The problem being is that for some supermarkets I got more answers than other supermarkets eg. supermarket 1 has 289 answers and supermarket 2 has 34 answers. Therefore the ranking is biased : supermarket 2 had less answers but has a better grade than supermarket 1, and supermarket 2 is ranked before supermarket 1 .
My question is do you know how to calibrate the data in order to get a grade which shows a better representation of the supermarket performance compared to the others ?
Thank you.
 A: More information about the survey and the difference in preferences
would be needed for a general answer to your question. However,
here is an example to use in your thinking about this discrepancy.
Consider one key summary question. "Overall, would you choose to shop
there again?" Suppose 189 out of 289 at market A said Yes and
26 out of 34 at market B said Yes. The proportions are 65% Yes for A
and 76% for B. But a test of proportions finds no significant difference.
Output from Minitab; P-value above 5%:
Test and CI for Two Proportions 

Sample    X    N  Sample p
1       189  289  0.653979
2        26   34  0.764706

Difference = p (1) - p (2)
Estimate for difference:  -0.110727
95% CI for difference:  (-0.263492, 0.0420386)
Test for difference = 0 (vs ≠ 0):  
  Z = -1.29  P-Value = 0.195

So it would be irresponsible to give B a higher rating than A.
Maybe an asterisk leading to a footnote, something like: 
"Too little data to rate.
Proportion of Yes's from a small number of responses is 76%.
Note: A major US consumer-oriented publication often prints numbered
lists of product or service rankings, along with a comment, such as: "Ratings must be separated
by about 5 positions on the list in order to be significantly different."
