2
$\begingroup$

This feels incredibly basic but I can't seem to find the answer!

Let's say I knew the pass rate for an exam in a huge population of students is 50% (let's say millions of students).

A new class of N=30 students take the test and 60% of the class pass. Another new class of N=5000 students take the test and 55% pass.

The test is binary, only pass or fail with no scores.

Is there a statistically sound way of "adjusting" both these classes towards the population average?

For instance, for the first class - although 60% passed there's only N=30, so this is adjusted towards the population mean by a lot (let's say 51%?)

The second class has a 55% pass rate but given there's N=5000 there will be only a small adjustment (maybe 54.9%)?

This makes sense in my head but I can't seem to find the right statistical approach!

The reason I want to do this is, if we found out there was an unmarked test from one student from both these new classes - what would we predict their chances of passing the test to be?

Many Thanks

$\endgroup$
6
  • $\begingroup$ Yes, there is, but you need to know the score distributions, or at the least the variances, as well. For example, in the first class, assume all 30 students got between a 59 and 61; you probably would guess the unmarked test should have somewhere between a 59 and 60. If, on the other hand, the scores were roughly uniformly distributed between 20 and 100, you might guess the unmarked test had a score somewhere between 50 and 60; there are formal ways of doing this calculation, but this should serve as an example. $\endgroup$
    – jbowman
    Jun 11 at 20:07
  • $\begingroup$ It's the formal calculations I'm interested in, I can't seem to find anything pointing at specific methods. Also, the example I gave is about test scores - but actually I'm looking at "success/fail", i.e. 60% of the class passes & 40% fail, which changes it slightly to your example. I'll edit the question to make this clearer. $\endgroup$
    – JackWills
    Jun 11 at 20:49
  • 2
    $\begingroup$ This sort of problem is addressed by Bayes or empirical Bayes methods, but it is not solvable with the information that you have given so far in your question. To present a solution, one would also need to know the heterogeneity between different classes in the past. You need to know not only the historical mean pass rate but also the historical variation between individual classes. It is the latter which allows one to determine the optimal weighting between the historical mean and the individual class mean. $\endgroup$ Jun 11 at 22:15
  • 1
    $\begingroup$ The most common solution for this sort of problem would be to setup a hierarchical beta-binomial model. If you have a historical record of results from a large number classes (you need the test result in each class, not just the population mean), then the beta prior for class successes rate can be estimated from the historical record. Estimating the prior in this way is called empirical Bayes. Once the beta prior is established, then each new class result yields a posterior beta distribution for the success rate specific to that class. $\endgroup$ Jun 11 at 23:27
  • $\begingroup$ Fantastic - we do have each classes pass rate & volume that forms the population mean so this seems doable. I'll do some research into the method above. Any resources appreciated! $\endgroup$
    – JackWills
    Jun 12 at 9:40

1 Answer 1

3
$\begingroup$

As noted in the comments, you need to the distribution of class pass rates in the population, not just the average over classes. This then becomes a Bayesian inference problem, where the prior is the distribution across classes, and the data is the number of passes and fails in a particular class.

As a simple model, let's say that pass rates are Normally distributed in the population with a mean of $\mu = .5$ and a standard deviation of $\sigma = .1$ (this is only an approximation, because this distribution doesn't rule out pass rates $<0$ or $>1$). You have data from a single class, where 15/20 students have passed. Let $\theta$ be the estimated underlying probability of passing for the class in question.

There are many ways to fit a model like this, but here's an example using rstan, which shows that the posterior estimate for the pass rate is 61%, with a 95% credible interval of [46% - 76%] (try changing the data or priors here to see how the results differ)

library(rstan)
model_code = '
data {
 real prior_mean;
 real prior_sd;
 int n_pass;
 int n_total;
}
parameters{
  real pass_rate;
}
model {
  pass_rate ~ normal(prior_mean, prior_sd);
  n_pass ~ binomial(n_total, pass_rate);
}'

data = list(
  prior_mean = .5,
  prior_sd = .1,
  n_pass = 15,
  n_total = 20
)
result = stan(model_code = model_code, data = data)  # Takes a few seconds to compile
print(result)

#   Inference for Stan model: 51931e3aee4b65fffc40e86f3ba918b6.
# 4 chains, each with iter=2000; warmup=1000; thin=1; 
# post-warmup draws per chain=1000, total post-warmup draws=4000.
# 
#             mean se_mean   sd   2.5%    25%    50%    75%  97.5%
# pass_rate   0.61    0.00 0.08   0.46   0.56   0.61   0.66   0.76
# lp__      -13.24    0.02 0.72 -15.25 -13.41 -12.96 -12.78 -12.73
# n_eff Rhat
# pass_rate  1473    1
# lp__       1534    1
# 
# Samples were drawn using NUTS(diag_e) at Sun Jun 12 14:33:35 2022.
# For each parameter, n_eff is a crude measure of effective sample size,
# and Rhat is the potential scale reduction factor on split chains (at 
# convergence, Rhat=1).
```
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.