4
$\begingroup$

I have a subject who was giving 2-alternative forced choice responses for a task consisting of 80 trials. For each trial, there was a 50% chance for each of the two options to be correct. This means chance-level responding would, overall, be linked to a 50% mean accuracy across all trials. But since no subject's average accuracy will typically be precisely 50%, I am wondering what statistical concept allows me decide on a cut-off value for how far from 50% the mean accuracy would have to be to decide the subject was not responding randomly.

I know the binomial distribution can be used to model such responses, but I am not sure at which outcome I should look at. For instance, this page computes the binomial and cumulative probabilities associated with an experimental session of a certain

  • Probability of success on a single trial
  • Number of trials
  • Number of successes (x)

My subject responded correctly on 51 out of the 80 trials, i.e. 63.75% correct. This gives a Cumulative Probability of P(X < 51)=0.99. How should I interpret this Cumulative Probability in plain English, in order to answer the question of whether or not the subject was responding randomly (at chance)? Would it not be quite arbitrary to decide, based on such a distribution, whether or not he was in fact understanding and doing his best at the task?! It's not like a simple % mean accuracy corresponds, 1-to-1, to a level of understanding of / involvement in the task!

Thanks!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If the cumulative probability is $99\%$, then there is only a $1\%$ chance that you would have seen a statistic at least as large as $51$. This suggests that the probability is >50%.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Sorry, still not clear on this. What the calculator refers to as 'cumulative probability' is in fact two different and (probably) complementary metrics: Cumulative Probability: P(X < 51), which is ~0.99, and Cumulative Probability: P(X > 51), which is ~0.04. Also, not sure which probability you refer to when you say "This suggests that the probability is >50%.". Are you saying I can infer from this data that the subject was NOT answering by chance? What about my comment about the arbitrary decision line defining what we think means answering by chance (not understanding/caring about the task)? $\endgroup$
    – z8080
    Commented Sep 15, 2014 at 9:47
  • 2
    $\begingroup$ @wildetudor the two measured are not strictly complementary, and in fact I am having a hard time reconciling the two: $P(X<51)+P(X>51)$ should be less than 1, yet its not, and we are still missing $P(X=51)$, so something unusual is going on with those numbers. As for your second question: you are trying to determine if the subject is answering randomly, so I don't see what is arbitrary about this -- its a straightforward test of proportion. In your case, it appears that your result is 2.5 standard deviations from the mean under the null...so its significantly different. $\endgroup$
    – user31668
    Commented Sep 15, 2014 at 12:54
  • $\begingroup$ I see. Just to be sure, the null in this case would be "the subject is answering randomly", in other words the statistic (here: cumulative distribution) is extreme enough that the null can be rejected, i.e. the subject is NOT answering randomly. Right? $\endgroup$
    – z8080
    Commented Sep 15, 2014 at 14:28
  • 1
    $\begingroup$ @wildetudor correct $\endgroup$
    – user31668
    Commented Sep 15, 2014 at 14:35
  • 1
    $\begingroup$ @wildetudor what exactly is contradictory? We've gone pretty far astray of the original question. The bottom line is that if you assume the person is guessing, then what is the probability of getting their result? This is a two tailed test, so you want the probability that they deviate from 50% by more than X (in either direction). Your use of the cumulative probability implicitly assumes an upper-tailed test, where you only care if the person is more than 50% accurate. $\endgroup$
    – user31668
    Commented Oct 14, 2014 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.