I am developing a composite score that pulls together multiple measures into one score for each group. Each measure is constructed as a percent (20% of customers met the measure, 60% of customers met the measure, etc.) While each measure is a total population, not a sample (e.g., all customers who bought product A, all customers who received service B, etc.), the denominators are all different. And some of those denominators may overlap (e.g., one customer could buy product A and receive service B).
I've been trying to figure out the best way to 1) make sure I have adequate sample sizes for each individual measure, and 2) determine if the results are statistically significantly different in aggregate.
I have run through multiple methods of doing this, and keep coming up with serious drawbacks.
- 20% of 1000 customers met measure 1
- 40% of 150 customers met measure 2
- 60% of 500 customers met measure 3
- 50% of 200 customers met measure 1
- 30% of 100 customers met measure 2
- 40% of 10 customers met measure 3
Measure 1 is worth 50%, Measure 2=30%, Measure 3=20%
Site A score: 34
Site B score: 42
Obviously, 40% of 10 customers is much more dubious than 60% of 500 customers.
The ultimate application: if an individual measure's population is too small to be reliable, I want to remove it. I want to judge sites based on whether they are significantly better or worse than the mean total score. But that mean total score is going to be based not just on one population, but multiple mini-populations, making it more difficult to weight.
I've been playing with a lot of potential solutions, but I'd love to hear any ideas!