How large sample size for defining difficulty I am currently making a website in which you can get several trivia questions. One thing I would like to implement is difficulty of questions. However, you can't just take a question and decide for yourself whether this questions is hard, easy, or medium. Because if you know it, well, then it's easy, and if not, it's probably hard.
So my idea was to make people push a yes or no button, i.e. did they know the answer for this question or not, save it to a data base, and then at some point (when the count was high enough for that particular question), then I could do something like:
if (count_yes + count_no) > #some_number:
    if (count_no / (count_yes + count_no)) < 0.333:
        difficulty = "easy"
    elif (count_no / (count_yes + count_no)) > 0.666:
        difficulty = "hard"
    else:
        difficulty = "medium"

return difficulty

Yes, this is just a simple Python code illustrating what I want to do.
So, my main question is: How large should the #some_number, i.e. sample size be, in order to achieve a somewhat realistic assumption of the difficulty ?
I mean, a sample size of 3 would not be enough, that's for sure. So is there some way to calculate or figure this out, or...?
 A: I'll be a little liberal interpreting your question. You're asking about question difficulty which we will frame as "percent of people that think this question is difficult".
This is a binomial problem, people either answer that a question is difficult or not. Essentially what we want to do is make sure our "difficulty" estimation is accurate. It's important to understand that you set the bounds of what is "accurate". If you say "I want the estimate to be within 25%" then we can do that with a small sample, if you want the estimate to be within .01%, then it will take a lot larger sample. You get to decide what works for your situation.
We need 2 things to estimate sample size in this problem, our desired alpha and our desired margin of error. Alpha is commonly to set to .05 which indicating that if our assumptions are correct and we ran this entire experiment many times, the true population proportion would lie outside our estimated confidence interval 5% of the time.
If you have more information we can change p in the formula to be .3 or .7 or whatever. Our formula is (assuming alpha=.05 [the 1.96 in the formula is the critical z score for alpha = .05]):
\begin{align} n &=\dfrac{p\cdot(1-p)\cdot1.96^2}{M^2}\\ \end{align}
Where M is the margin of error for our estimation. And p is the chance they say the question is difficult.
The most conservative we can be about estimating the number of samples needed is by assuming that 50% of people will answer that a question is difficult. So we can set p = .5
So if we want to estimate difficulty within 5% we get:
\begin{align} n &=\dfrac{0.5\cdot0.5\cdot1.96^2}{0.05^2}\\  &=384.16\\ \end{align}
For 10% we get:
\begin{align} n &=\dfrac{0.5\cdot0.5\cdot1.96^2}{0.1^2}\\  &=96.04\\ \end{align}
You can play around with the numbers yourself or dive a little more into the background here: https://onlinecourses.science.psu.edu/stat506/node/11/
