How do I determine/compute a cutoff for chance level/not chance level? I am in the process of describing my research design for my dissertation and ran into a roadblock. In my design, I am converting 20 y/n responses from 190 participants to two dichotomous groups: 1). Chance level and 2). Above chance level.
If chance level is 50% or 10 responses correct/incorrect, how do I determine what would not be chance? How do I determine the cutoff for placing an individual into the above chance level group?
I have heard a few things such as 25% over chance is no longer chance. But even here, I am not sure how to calculate it...is 50% the base or is 100% the base?
I could really use your help with this matter.
Thank you for your consideration.
======================================================================================
Hi Glen,
I asked a question about probabilities and cut-offs, to which you replied with a great answer. I appreciate the table you provided as it gives me a choice. However, now my dilemma is that I would like to know the source so I can cite it. My mentor wants to know my source!
Skootz
 A: Here's one possibility:
If you assume the answers are like independent coin-flips, you can work out the probability of getting any number correct. For example if the answers were 50-50 coin flips the change of getting at least 15 correct is about 2%.  You might say "well, that's pretty unlikely, let's say that 15 or more is 'above chance' since there's such a low probability of it happening by chance".
Indeed, that's how hypothesis tests are constructed.
Here's a table of values so you can choose which cutoff appeals to you:
Number    Probability of 
correct   at least that 
          many correct (%)
  13       13.1588
  14        5.7659
  15        2.0695
  16        0.5909
  17        0.1288
  18        0.0201
  19        0.0020
  20        0.0001

I'd suggest choosing one of 14, 15 or 16.
A: @Glen_b offered one solution. It is correct for each individual, and that's fine. But you have 190 individuals. By chance, some of them will be above chance level and some below. For example, here is a R code for a single case of 190 people flipping 20 coins.
set.seed(19291010)

choices <- rbinom(190, 20, .5)
table(choices)

If we say that 1 = heads and 0 = tails then 8 people got 14 heads, 4 got 15 heads and 1 got 16 heads; also, 9 got 14 tails, 3 got 15 tails and 1 got 4 tails. This makes sense, since, per Glen's table, 6 \% should get at least 14, and 6\% of 190 is 12 (with some rounding). 
This gets a little bit Bayesian. If you assume that the choices are random, then, by definition, anyone who gets a lot correct is just "lucky" (whatever that means!). But if it's really random, then in the next set of 20 tosses, mostly different people will get a lot of heads. 
If you assume that the choices are all skill, then anyone who gets a large number correct is "skillful" (or knowledgeable etc).
If you don't know if it's luck or skill, then you might compare the first 10 questions to the second 10; if there is more than chance consistency, that person is "good". But then you  have to figure out what the consistency will be, by chance. 
