Binomial Events with unequal probabilities Imagine I have a database of 1000 students. I test these students every week with questions that can be true/false. How do I rank the students according to the historical track records?
The problems are: 


*

*Each student has answered a different number of questions in the past. Many have only answered a few, others have answered over 50. 

*The questions are of varying difficulty. Sometimes the answer is obvious and other times it really is a 50:50 guess. Imagine I have a difficulty score for each question ranging from 1 - 100 (100 meaning very difficult).


How do I find the students who are truly geniuses, and separate them from the students who only perform at chance level (or only slightly above)?
Am I right to think that these are binomial events with unequal probabilities? 
 A: You should take a look at Item Response Theory.  This has long been an issue with tests where you have questions of different difficulties and test takers of different abilities. There are different approaches to estimating the abilities depending on what assumptions you are willing to make and what questions you want answered.
A: I don't think there's a single answer to that one.  For example, is a wrong answer equivalent to failing to answer?  If one is worse that the other, how much worse?  
I would not call the distribution binomial because there are three possibilities:  correct, wrong, and fail to answer.  If differences in difficulty were not an issue you'd have a multinomial distribution, but it sounds like differences in difficulty are important so you now need some way to assign weights that are a function of difficulty.
I suspect that if you ask five people to each come up with a scoring system you'll get more than five different systems and different students may object to different systems.
If all you want is to find the "geniuses", start with the ones who ask good questions, have discussions with them and use your intuition.
A: You need a model along with assumptions. I'll propose a simple one. Let a student's score on a question be the difficulty of the question if he/she answers correctly, and 0 otherwise. Sum up each student's scores and divide by the number of questions the student has attempted. After doing this for all students, compile the data and look at the distribution. Choose a threshold that means "genius" to you and define the students who score above that threshold as geniuses.  
Note that there are several issues you will have to deal with based on this approach. For example, how you handle students who have answered only a small number of questions. 
To answer the question at the end of your post, yes, I think you are right in your belief that the students can be viewed as having differing binomial success probabilities. 
