I think we have all taken exams where as you answer more questions correctly, the exam gets harder. Intuitively, this is obvious why...If an IQ test had questions for 3rd graders, everyone would get the same answer, so by making the exam harder, you can differentiate better between different participants.
However, rigorously speaking, assuming the goal is to differentiate participants, is it best to make the exam get harder until each answers correctly 50% of the time? This seems intuitively true – but if so, why? I have been trying to attempt to answer this question using entropy, KL divergence, etc., but I haven't been able to find precisely the right framework for thinking about this question.
What makes this question challenging for me is that we want to quantify: "How easy is it to tell if someone is at the wrong (or right) value for the difficulty?" But we also want to quantify this individual value of difficulty compares to the population of true difficulties, $d_i$. Because clearly if someone has an innate ability $d_k$, then a difficulty $d<<d_k$ will quickly tell you that someone is not at difficulty $d$, but this does not give you much information because that will be true for most participants.
Perhaps this is overcomplicating things, but I've been trying to concretize this problem as follows: for difficulties $d$, there is a mapping $f(d)\rightarrow p\in[0,1], f(\infty)=1, f(0)=.5, f(\infty)=0 $, where $p = Pr($answering correctly $)$. Each person has an ability $d_i$, corresponding to the mapping $f_i(d) =f(d-d_i)$. What can we say about this system to effectively distinguish individuals' $d_i$?