How do you determine the weight of different sample sizes? I'm not intimately familiar with statistics, please keep your answers in the realm a layman could understand.
I've been tasked to create rankings for QA scores at a company. I could just use the average scores and throw that up on the board. However, that does not seem correct.
If I have someone that has had 200 QA's and averages out at 80%, and another person who has had 5 QA's and averages out at 85%, it does not seem like a fair comparison to rank person #2 higher. A single high-QA could bump his score up significantly where the same would hardly touch the first persons overall average.
What would be a good way to weight these? Do these need to have weights based on total number of QAs?
 A: Checking with the standard reference on weights (https://stats.idre.ucla.edu/other/mult-pkg/faq/what-types-of-weights-do-sas-stata-and-spss-support/), I think that you have a legit case for analytic weights based on the underlying sample size. That is to say, you can average the QA scores with weights proportional to the number of QAs by a given rater (if you believe in variances inversely proportional to the sample size, or precision which is the inverse of variance), or to the square root of that number (if you believe in standard deviations of the sample mean).
Another approach could be to test the hypothesis that one rater is better than the other. Rather than forming a frequentist test for proportions, you may be better off with a Bayesian approach with a prior that the two raters have the same proportions, and evaluating the posteriors with the answers they provided.
I will modify the numbers in your example, so that it works like a "hit" vs "no hit". So let's say the first guy did 200 QAs with 150 hits and 50 no hits at 75% rate, and the second guy, 5 QAs with 4 hits and 1 no hit. Denoting the probability of a hit by the $i$-th rater as $\theta_i$, let's start with a uniform prior:
$$
p(\theta_1) = U[0,1], p(\theta_2) = U[0,1], \theta_1 \perp \theta_2
$$
For this model, 
$$
{\rm Pr\,} [\theta_1 < \theta_2] = \int_{0}^{1} \int_0^{\theta_2} {\rm d}p(\theta_1) \, {\rm d} p(\theta_2) = 1/2
$$
which is the area of a triangle above the diagonal -- no big deal, we don't know anything about the raters.
Let us now invoke conjugacy of binomial with the beta distribution (using the fact that the prior was $U[0,1] \equiv B(1,1)$). Thus with $k_i$ successes in $n_i$ trials, the posterior for each $\theta_i$ is $B(k_i+1, n_i-k_i+1)$:
$$
p(\theta_1|\mbox{QA data}) = B(161,41), \quad p(\theta_2 | \mbox{QA data}) = B(5,2)
$$
Now, with the data at hand, we can evaluate
$$
{\rm Pr\,} [\theta_1 < \theta_2] = \int_{0}^{1} \int_0^{\theta_2} {\rm d}p(\theta_1|\mbox{QA data}) \, {\rm d} p(\theta_2|\mbox{QA data}) = 
$$
$$
= \int_{0}^{1} \int_0^{\theta_2} \frac{x^{k_1}(1-x)^{n_1-k_1}}{B(k_1+1,n_1-k_1+1)} {\rm d}x \, {\rm d} p(\theta_2|\mbox{QA data}) =
$$
$$
= \int_{0}^{1} I_{\theta_2}(k_1+1,n_1-k_1+1) \, {\rm d} p(\theta_2|\mbox{QA data})
$$
$$
= \int_{0}^{1} I_{x}(k_1+1,n_1-k_1+1) \frac{x^{k_2+1} (1-x)^{n_2-k_2+1}}{B(k_2+1,n_2-k_2+1)} \, {\rm d} x
$$
where $I_x(a,b)$ is the incomplete Beta function. This would have to be integrated numerically:
## k1 <- 150
## n1 <- 200
## k2 <- 4
## n2 <- 5
## 
## integrate( 
##  f = function(x) { pbeta(x,k1+1,n1-k1+1)*dbeta(x,k2+1,n2-k2+1) },
##  lower=0,upper=1
## )
0.4704129 with absolute error < 1.7e-06

So there is very, very weak evidence that rater 1 is better than rater 2, so I would reluctantly put the first guy over the second guy. If we go up to 8 successes out of 10 trials for rater 2, though, the conclusion is reversed, albeit the evidence is still weak:
## k1 <- 150
## n1 <- 200
## k2 <- 8
## n2 <- 10
## 
## integrate( 
##  f = function(x) { pbeta(x,k1+1,n1-k1+1)*dbeta(x,k2+1,n2-k2+1) },
##  lower=0,upper=1
## )
0.5485611 with absolute error < 2.5e-06

A: The most intuitive way to develop a score is to sum each score. This way, if two people have an average of a score of 80 but one person has ten QAs and another person has a hundred, then their scores would be 800 and 8,000, respectively. A potential issue with this is that a lot of bad scores can outweigh a few good scores. If someone has ten perfect 100 scores, that is the same as someone else having 1,000 scores of 1. Without looking at your data, I don't know how realistic these scenarios are.
What I would ultimately suggest is developing two scores: an average and a sum. You can then combine them and rank them. If the averages are considered, then someone who has many QAs but performs poorly won't sneak by as "great." If the sums are considered, then someone who has one or two rave QA reviews won't be overly considered. Rank the average and the sum separately. Thus each person will have two "rank" scores. You can take the average of these to give an aggregate score, then rank these to get your final ordering from top to bottom.
There may be a better way to do this, but this is how I would start the problem without reading lots of research.
