Adjusting for variance-sample size dependence

Question

In many survey results that are released to the public with respect to ranking (whether it be school ranking, city crime rates, etc.), one of the many criticisms of these is that some which have small sizes are not adjusted for sample size - hence, in such results, the highest-ranked and lowest-ranked entities often have an extremely small sample size.

I think it's reasonable to say that, typically, in such a survey, when scores are computed, units within the entity are given a score (say $N$ units with scores $X_i$), and the final score that an entity is some sort of a linear function of the $X_i$s: $$\text{Entity Score} = \dfrac{\sum_{i=1}^{N}X_i}{N}\text{.}$$ If we assume that the $X_i$s are independent and identically distributed with variance $\sigma^2$, you end up with the famous equation for the variance of an average: $\dfrac{\sigma^2}{N}$. Obviously, as $N$ increases, the variance decreases (holding $\sigma^2$ constant). When you're considering data where the size $N$ differs considerably among different entities and you're trying to rank these entities against each other, there is much more variation among schools with small $N$ over large $N$.

What I'm interested in knowing is how people adjust for such cases. Perhaps this question is too broad for this website, but I remember always hearing stats professors point these problems out, yet not provide any sort of solution, other than to be careful about how you interpret the statistics.

Furthermore, as much as I like standardizing by subtracting the mean and dividing by some form of standard deviation, the problem with this is that it's extremely hard to interpret to a layperson. With my current position in particular, it is necessary for the public to be able to replicate whatever calculations we make whenever possible, so having it at least be accessible to laypersons is extremely important. However, I welcome all answers, technical or not - but I would strongly prefer methods accessible to laypersons of statistics.

Answer 1

(Empirical) Bayesian approach: As @whuber mentions, I think the most natural approach is a Bayesian approach, or possibly even an empirical Bayesian approach.

In particular, let's call the true entity scores $e_j$ for $j=1,\dotsc,m$. To estimate these you have data $X_{j1},\dotsc, X_{jn_j}$ for each $j$. Note that $n_j$ is different in each case.

Now assume you have a prior $g$ for the $e_j$, i.e. $e_j \sim g$, then rather than estimating $e_j$ by the sample mean $\hat{e_j} = \frac{\sum_{i=1}^{n_j}X_{ij}} {n_j}$, you could take the posterior mean:

$$\tilde{e_j} = \mathbb E[e_j | X_{j1}, \dotsc, X_{jn_j}]$$

This approach would even work if you don't want to put a prior $g$ on these scores; instead you could learn the prior $g$ from your data. This is called Empirical Bayes. Bradley Efron recently wrote a paper on how to do this "almost" nonparametrically ("almost" because he does not do actual nonparametrics, but considers flexible exponential families with many parameters). For a simpler approach, David Robinson has a very nice blog post, where he elaborates on this idea based on the example on determining the best batters (i.e. rank based on average hits the players gets).

Ad-hoc frequentist approach: Another approach which in my opinion is a lot more ad-hoc, but has the advantage of being simpler and possibly easier to explain, is to use the lower bound of a ($1-\alpha$) confidence interval of the parameter for the ranking of these entity scores. For example, if $\hat{\sigma}_j$ is the standard deviation estimate based on the $n_j$ samples for the $j$-th entity score, then you could rank based on:

$$ \bar{e_j} = \hat{e_j} - \frac{\hat{\sigma}_j}{\sqrt{n_j}}z_{1-\frac{\alpha}{2}}$$

Here $z_{1-\frac{\alpha}{2}}$ is the $1-\frac{\alpha}{2}$ quantile of a standard Normal distribution (could use more elaborate schemes of course). This has the advantage of accounting for sample size directly and being simple to compute; but again I think it is quite ad-hoc. [There was a quite famous blog post by someone using such an approach for his internet company; I cannot find it right now unfortunately. Maybe someone that reads this can also point me to that post.]