9
$\begingroup$

I need help figuring out the correct way to calculate winners at our science fair. I don't want my ignorance of statistics & math to get in the way of a kid's chances of winning. (lots of scholarship & advancement benefits at stake). Thanks in advance for your help.

First a little background of how we have things set up:

Our fair typically has around 600 student projects. These projects are completed and presented by individual students or a team of students. A team can consist of 2 or 3 kids.

The students are divided into two divisions: Elementary (grades 6-8) and Secondary (grades 9-12). Each division has different categories: 9 categories for Elementary projects and 17 categories for the Secondary division projects.

Awards are given for first, second and third place for each category in each division. Honorable mention awards are also given for placements beyond third place.

For each project, we assign between 4 to 6 judges. We make our assignments based on the judges' qualifications, their category preference and their past judging experience. (more experienced are assigned to the senior division projects).

How the judges score a project:

For each project there are 5 criteria that are assigned points. Each criteria can be awarded between 1 and 20 points. General criteria are:

  • Overall objective + hypothesis + use of resources (1..20)
  • Design + procedures (1..20)
  • Data collection + results (1..20)
  • Discussion + conclusion (1..20)
  • Interview (1..20)

For team projects a sixth criteria is assessed called "team deduction", where a judge can deduct points (up to 15) for teammates who didn't participate or didn't show up.

  • Team deduction (0..-15)

So a judge can score every project between 5 and 100 points. If the project is a team project, the score can be reduced by 15 points.

Raw data:

During the course of a few hours we collect up to 3,600 scores from judges. These scores are entered into a database where I can do all kinds of sorting, averaging, standard deviation calculations, etc. I just don't know exactly what I should do with these raw scores. Right now, I'm doing a simple average for each project, but I worry that I'm not adjusting for judge biases, team deductions, or any number of other things that I'm not considering.

Desired result:

In the end, I'd like to process the scores so that I can award first, second and third place projects for each category, and then honorable mention awards for the subsequent places. I would like to be confident that the positions were calculated correctly and the kids who win are deserving of the recognition (and prizes).

Thanks a lot for reading my long question and for your help figuring this out. I'll be happy to answer any follow-up questions you may have.

$\endgroup$
2
  • 1
    $\begingroup$ Interesting and tough question and you've put your finger on some of the key issues. How many judges in total, so how many projects would each judge judge? (some idea of the range as well as average would be good). Also, are the 26 categories mutually exclusive? I have a hunch the best answer may be just to do the average for each project, but it might be possible to adjust for judges. I'll read the answers with interest! $\endgroup$ Jan 25 '12 at 9:24
  • $\begingroup$ Thanks for your initial interest. I've added more info to the answer below. If you have any insight, I'd appreciate your help. $\endgroup$
    – Mike Davie
    Jan 25 '12 at 20:03
2
$\begingroup$

I think that "answer" is possibly too generous a label for my thoughts here. I love exploratory data analysis, and I am a huge boxplot fan, so that is going to be reflected in my comments..

Hi, that's a lot of scores. :) It sounds like you have at least 78 projects out of the 600 getting in the top 3 ([9+17]x3) plus the honourable mentions. Normally I would say to sample from the top and middle of each category to conduct an audit of scoring, but that would be very onerous in your case because of the numbers you have - and it's just you finalising the scoring. :)

I'm hoping you might have a statistics package available to you, as I have some suggestions that you could use below.

Have you looked at the spread of scores within each category? Are the top 3, or 5, or 8 projects very close for scores? That would suggest that the quality of the projects is very similar and no matter what you do, there is probably going to be at least a perception of arbitrariness around the final scores.

I'm not sure how many projects each judge scores. Assuming they score a reasonable number (say >10, although the higher the better here), for each judge you could calculate the median and interquartile range for the total score given to each project they assess (you have so many attributes, it's probably not worth looking at each of them individually). Do any judges seem to be giving particularly high scores, or particularly low scores? Do any judges seem to be scoring consistently in the middle so they are possibly giving 10s, this can be shown by a comparatively small interquartile range and a total score median around the middle of the range of possible values.

For the team projects, you could compare their placing on the basis of total scores, to their placing once the team deduction has been applied. Are the team deductions affecting teams that would otherwise be in the top 3?

These are just suggestions to get you started. I think visualising the data along these lines would give you some good indicators about whether the placings seem fair.

Update: this is an interestingly difficult problem that you have. It sounds like each individual judge doesn't assess enough projects for us to be able to come up with a weighting factor for each judge (to take account of judge bias), because we don't have enough data to be able to measure inter-rater reliability across the judges, there just isn't enough overlap for judges scoring on the same projects to do that. Did you look at the score range for the top few projects - were there clear differences between them and lower-scoring projects (natural boundaries?), how close in score were the top projects?

Out of curiosity, were the judges given scoring criteria, so they had little flexibility in how to give scores on each criterion (e.g. give 1 point for providing a null hypothesis, give 1 point for providing one or more alternative hypotheses...) or were did they just know the total number of points they could award and the rest was left up to them? If they had a scoring guide, I would be more confident that the scores were reasonably accurate.

$\endgroup$
7
  • 1
    $\begingroup$ I too would be curious as to the spread of the scores - is there clearly some "top scores", or is there a clump and who comes out on the top of that is a bit...opaque in terms of process. Like the college admissions process these kids will experience later :) $\endgroup$
    – Fomite
    Jan 25 '12 at 4:11
  • $\begingroup$ Thanks Michelle for your thoughts. I really appreciate you taking the time. To answer your questions, judges only judge a small number of projects. We have minimums for each division: 4 times for Elementary and 5 or 6 times for Secondary (5 times for High School Juniors and 6 times for Seniors). $\endgroup$
    – Mike Davie
    Jan 25 '12 at 20:03
  • $\begingroup$ I need to clarify the above. The minimum scores are per project, not how many times the judges will score projects during the fair. A typical judge will judge anywhere between 8 and 15 projects during the fair. That number depends on the judges availability, their qualifications, willingness to help, etc. $\endgroup$
    – Mike Davie
    Jan 25 '12 at 20:17
  • 2
    $\begingroup$ Good answer Michelle. Some ideas from me: 1) Definitely give the judges some kind of rubric to try to encourage common standards; 2) try to have the same number of judges per project if possible (as otherwise the projects with fewer judges will have a higher variance and hence more chance of getting to the top - or the bottom) and 3) I think you will have to just use an average, but if you had the expertise and software you could fit a mixed effects model with judge as a random effect and it see if this changes the result. What if it does? Probably still use the average.... $\endgroup$ Jan 25 '12 at 22:05
  • $\begingroup$ Thanks again for everyone's input. After letting this knock around in my head for a while, I decided to look to see how projects are scored at the international level (one competition step above our fair). The international fair is called ISEF. We send our top 5 students to ISEF each year. $\endgroup$
    – Mike Davie
    Jan 27 '12 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.