In many survey results that are released to the public with respect to ranking (whether it be school ranking, city crime rates, etc.), one of the many criticisms of these is that some which have small sizes are not adjusted for sample size - hence, in such results, the highest-ranked and lowest-ranked entities often have an extremely small sample size.

I think it's reasonable to say that, typically, in such a survey, when scores are computed, units within the entity are given a score (say $N$ units with scores $X_i$), and the final score that an entity is some sort of a linear function of the $X_i$s: $$\text{Entity Score} = \dfrac{\sum_{i=1}^{N}X_i}{N}\text{.}$$ If we assume that the $X_i$s are independent and identically distributed with variance $\sigma^2$, you end up with the famous equation for the variance of an average: $\dfrac{\sigma^2}{N}$. Obviously, as $N$ increases, the variance decreases (holding $\sigma^2$ constant). When you're considering data where the size $N$ differs considerably among different entities and you're trying to rank these entities against each other, there is much more variation among schools with small $N$ over large $N$.

What I'm interested in knowing is how people adjust for such cases. Perhaps this question is too broad for this website, but I remember always hearing stats professors point these problems out, yet not provide any sort of solution, other than to be careful about how you interpret the statistics.

Furthermore, as much as I like standardizing by subtracting the mean and dividing by some form of standard deviation, the problem with this is that it's extremely hard to interpret to a layperson. With my current position in particular, it is necessary for the public to be able to replicate whatever calculations we make whenever possible, so having it at least be accessible to laypersons is extremely important. However, I welcome all answers, technical or not - but I would strongly prefer methods accessible to laypersons of statistics.

  • 3
    $\begingroup$ +1 This is an important question that recurs frequently, often in the guise of ranking recommendations. Recently, in a comment thread, I made a tiny initial stab at suggesting a principled strategy. It considers prior information, the different variances, and a loss function associated with the decision (which is a ranking of the entities in this case). $\endgroup$
    – whuber
    Mar 24, 2016 at 20:20

1 Answer 1


(Empirical) Bayesian approach: As @whuber mentions, I think the most natural approach is a Bayesian approach, or possibly even an empirical Bayesian approach.

In particular, let's call the true entity scores $e_j$ for $j=1,\dotsc,m$. To estimate these you have data $X_{j1},\dotsc, X_{jn_j}$ for each $j$. Note that $n_j$ is different in each case.

Now assume you have a prior $g$ for the $e_j$, i.e. $e_j \sim g$, then rather than estimating $e_j$ by the sample mean $\hat{e_j} = \frac{\sum_{i=1}^{n_j}X_{ij}} {n_j}$, you could take the posterior mean:

$$\tilde{e_j} = \mathbb E[e_j | X_{j1}, \dotsc, X_{jn_j}]$$

This approach would even work if you don't want to put a prior $g$ on these scores; instead you could learn the prior $g$ from your data. This is called Empirical Bayes. Bradley Efron recently wrote a paper on how to do this "almost" nonparametrically ("almost" because he does not do actual nonparametrics, but considers flexible exponential families with many parameters). For a simpler approach, David Robinson has a very nice blog post, where he elaborates on this idea based on the example on determining the best batters (i.e. rank based on average hits the players gets).

Ad-hoc frequentist approach: Another approach which in my opinion is a lot more ad-hoc, but has the advantage of being simpler and possibly easier to explain, is to use the lower bound of a ($1-\alpha$) confidence interval of the parameter for the ranking of these entity scores. For example, if $\hat{\sigma}_j$ is the standard deviation estimate based on the $n_j$ samples for the $j$-th entity score, then you could rank based on:

$$ \bar{e_j} = \hat{e_j} - \frac{\hat{\sigma}_j}{\sqrt{n_j}}z_{1-\frac{\alpha}{2}}$$

Here $z_{1-\frac{\alpha}{2}}$ is the $1-\frac{\alpha}{2}$ quantile of a standard Normal distribution (could use more elaborate schemes of course). This has the advantage of accounting for sample size directly and being simple to compute; but again I think it is quite ad-hoc. [There was a quite famous blog post by someone using such an approach for his internet company; I cannot find it right now unfortunately. Maybe someone that reads this can also point me to that post.]

  • $\begingroup$ Do you have a recommendation on texts covering Empirical Bayes? I have Doing Bayesian Data Analysis by Kruschke and the text by Gelman et al. but I'm pretty sure neither of them cover it. $\endgroup$ Mar 24, 2016 at 21:52
  • $\begingroup$ @Clarinetist, hmm.. I think Efron's "Large-Scale Inference: Empirical Bayes Methods" is your best bet. It is a very clearly written and illuminating book. One problem is that it is mainly focused on multiple testing rather than on estimation, but the 1st chapter deals with estimation and iirc also has that baseball example. The original "baseball" papers could also be useful. Maybe the following video by Rafael Irizarry could also be helpful: youtube.com/watch?v=QINX3cI7qgk $\endgroup$
    – air
    Mar 24, 2016 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.