I need help figuring out the correct way to calculate winners at our science fair. I don't want my ignorance of statistics & math to get in the way of a kid's chances of winning. (lots of scholarship & advancement benefits at stake). Thanks in advance for your help.
First a little background of how we have things set up:
Our fair typically has around 600 student projects. These projects are completed and presented by individual students or a team of students. A team can consist of 2 or 3 kids.
The students are divided into two divisions: Elementary (grades 6-8) and Secondary (grades 9-12). Each division has different categories: 9 categories for Elementary projects and 17 categories for the Secondary division projects.
Awards are given for first, second and third place for each category in each division. Honorable mention awards are also given for placements beyond third place.
For each project, we assign between 4 to 6 judges. We make our assignments based on the judges' qualifications, their category preference and their past judging experience. (more experienced are assigned to the senior division projects).
How the judges score a project:
For each project there are 5 criteria that are assigned points. Each criteria can be awarded between 1 and 20 points. General criteria are:
- Overall objective + hypothesis + use of resources (1..20)
- Design + procedures (1..20)
- Data collection + results (1..20)
- Discussion + conclusion (1..20)
- Interview (1..20)
For team projects a sixth criteria is assessed called "team deduction", where a judge can deduct points (up to 15) for teammates who didn't participate or didn't show up.
- Team deduction (0..-15)
So a judge can score every project between 5 and 100 points. If the project is a team project, the score can be reduced by 15 points.
Raw data:
During the course of a few hours we collect up to 3,600 scores from judges. These scores are entered into a database where I can do all kinds of sorting, averaging, standard deviation calculations, etc. I just don't know exactly what I should do with these raw scores. Right now, I'm doing a simple average for each project, but I worry that I'm not adjusting for judge biases, team deductions, or any number of other things that I'm not considering.
Desired result:
In the end, I'd like to process the scores so that I can award first, second and third place projects for each category, and then honorable mention awards for the subsequent places. I would like to be confident that the positions were calculated correctly and the kids who win are deserving of the recognition (and prizes).
Thanks a lot for reading my long question and for your help figuring this out. I'll be happy to answer any follow-up questions you may have.