How to fairly determine winners for a regional science fair?

Question

I need help figuring out the correct way to calculate winners at our science fair. I don't want my ignorance of statistics & math to get in the way of a kid's chances of winning. (lots of scholarship & advancement benefits at stake). Thanks in advance for your help.

First a little background of how we have things set up:

Our fair typically has around 600 student projects. These projects are completed and presented by individual students or a team of students. A team can consist of 2 or 3 kids.

The students are divided into two divisions: Elementary (grades 6-8) and Secondary (grades 9-12). Each division has different categories: 9 categories for Elementary projects and 17 categories for the Secondary division projects.

Awards are given for first, second and third place for each category in each division. Honorable mention awards are also given for placements beyond third place.

For each project, we assign between 4 to 6 judges. We make our assignments based on the judges' qualifications, their category preference and their past judging experience. (more experienced are assigned to the senior division projects).

How the judges score a project:

For each project there are 5 criteria that are assigned points. Each criteria can be awarded between 1 and 20 points. General criteria are:

• Overall objective + hypothesis + use of resources (1..20)
• Design + procedures (1..20)
• Data collection + results (1..20)
• Discussion + conclusion (1..20)
• Interview (1..20)

For team projects a sixth criteria is assessed called "team deduction", where a judge can deduct points (up to 15) for teammates who didn't participate or didn't show up.

• Team deduction (0..-15)

So a judge can score every project between 5 and 100 points. If the project is a team project, the score can be reduced by 15 points.

Raw data:

During the course of a few hours we collect up to 3,600 scores from judges. These scores are entered into a database where I can do all kinds of sorting, averaging, standard deviation calculations, etc. I just don't know exactly what I should do with these raw scores. Right now, I'm doing a simple average for each project, but I worry that I'm not adjusting for judge biases, team deductions, or any number of other things that I'm not considering.

Desired result:

In the end, I'd like to process the scores so that I can award first, second and third place projects for each category, and then honorable mention awards for the subsequent places. I would like to be confident that the positions were calculated correctly and the kids who win are deserving of the recognition (and prizes).

Thanks a lot for reading my long question and for your help figuring this out. I'll be happy to answer any follow-up questions you may have.

Answer 1

I think that "answer" is possibly too generous a label for my thoughts here. I love exploratory data analysis, and I am a huge boxplot fan, so that is going to be reflected in my comments..

Hi, that's a lot of scores. :) It sounds like you have at least 78 projects out of the 600 getting in the top 3 ([9+17]x3) plus the honourable mentions. Normally I would say to sample from the top and middle of each category to conduct an audit of scoring, but that would be very onerous in your case because of the numbers you have - and it's just you finalising the scoring. :)

I'm hoping you might have a statistics package available to you, as I have some suggestions that you could use below.

Have you looked at the spread of scores within each category? Are the top 3, or 5, or 8 projects very close for scores? That would suggest that the quality of the projects is very similar and no matter what you do, there is probably going to be at least a perception of arbitrariness around the final scores.

I'm not sure how many projects each judge scores. Assuming they score a reasonable number (say >10, although the higher the better here), for each judge you could calculate the median and interquartile range for the total score given to each project they assess (you have so many attributes, it's probably not worth looking at each of them individually). Do any judges seem to be giving particularly high scores, or particularly low scores? Do any judges seem to be scoring consistently in the middle so they are possibly giving 10s, this can be shown by a comparatively small interquartile range and a total score median around the middle of the range of possible values.

For the team projects, you could compare their placing on the basis of total scores, to their placing once the team deduction has been applied. Are the team deductions affecting teams that would otherwise be in the top 3?

These are just suggestions to get you started. I think visualising the data along these lines would give you some good indicators about whether the placings seem fair.

Update: this is an interestingly difficult problem that you have. It sounds like each individual judge doesn't assess enough projects for us to be able to come up with a weighting factor for each judge (to take account of judge bias), because we don't have enough data to be able to measure inter-rater reliability across the judges, there just isn't enough overlap for judges scoring on the same projects to do that. Did you look at the score range for the top few projects - were there clear differences between them and lower-scoring projects (natural boundaries?), how close in score were the top projects?

Out of curiosity, were the judges given scoring criteria, so they had little flexibility in how to give scores on each criterion (e.g. give 1 point for providing a null hypothesis, give 1 point for providing one or more alternative hypotheses...) or were did they just know the total number of points they could award and the rest was left up to them? If they had a scoring guide, I would be more confident that the scores were reasonably accurate.