# Calculating the probability of being correct due to chance

I'm looking for a third opinion on the probability of making a particular decision due to chance alone.

I have a task where people read a paragraph of text, then they are shown it a second time, except some of those times one word has changed. People are then asked whether they detected a change. If they say no then that's the end of the task. If they say yes, then they are given a set of nine words and are asked to respond which word was changed.

If people correctly guess that a change has occurred AND which word was changed, then a single "correct" is registered. If they correctly detect a change and wrongly guess the word that has changed then an "incorrect" is registered (similarly, it is incorrect if a word has changed but the person did not detect it).

What I'm trying to work out is what is the probability of correctly answering that a word has change, AND which word it was, due to chance alone. My first thought was that it was 1/18. That's a 1/2 chance of firstly correctly guessing that a change had occurred, followed by a 1/9 chance of correctly guessing the word.

Someone I've spoken to has suggested that the chance rate is actually 1/10. In effect, "no, a change has not occurred" becomes the tenth option.

So, is the chance of being correct 1/18, 1/10, or are we both wrong?

• This question appears to offer insufficient information for an answer. On what basis are we to compute a probability for correctly guessing that a word has changed? Are you assuming that this guess is made by a mechanism tantamount to flipping a fair coin? If so, why is that a reasonable model for "chance" in this experiment?
– whuber
Aug 25, 2013 at 20:01
• To elaborate on @whuber's answer, your question indicates three levels of probabilities: a) whether the text shown for the second time actually constitutes a changed version or not (you write "except some of those times one word has changed"). b) whether the person detects a change c) what is the probability that it picks the correct changed word. It would help you to draw the relevant event-and-probability tree. And what if he is shown an unchanged version, says "yes there is a change", and then picks a word that he (incorrectly) thinks it has changed? Aug 25, 2013 at 21:25
• Thanks for the responses. You're correct that whether or not the text actually changes factors into the probability, but, that withstanding, I'm still not entirely sure what to do in the cases where the person says that no change has occurred (regardless of whether it has or hasn't). In terms of a tree, it seems to me that if someone says no change then there should be only be one branch (1/1), and if they say yes then there should be 9 branches (each 1/9). By that logic, I get 1/18 (if we forget for a second that sometimes a change doesn't occur). But my second opinion was that it should 1/10 Aug 25, 2013 at 22:25
• You cannot validly apply "logic" without first positing a probability model. That's the point of my first comment and that of @Alecos. As long as you keep trying to brush that issue aside, you will have no rational basis for settling your question. What troubles me even more, though, is that this question has the appearance of trying to formulate a statistical hypothesis test--but it is difficult to see what validity or use it would have. This compels me to inquire why you want to assess these probabilities in the first place: what's the point?
– whuber
Aug 26, 2013 at 15:19
• From my point of view at least, if you say that "a change happens half of the time" you do exactly what @whuber says - you posit a probability model (you assume the probability distributions). The other part is the structure of available choices, which I outlined in my answer. So it is perfectly relevant, but it is what you are actually doing -but it is important for one to rememeber that the assumptions like "change happens half of the time" are not made to "keep the math a little easier", but they have critical consequences on the results one will obtain. Aug 26, 2013 at 18:31

Let $p$ be the probability that we show a changed version. Let $q$ be the probability that the person says "yes there is a change". Let $v_1,...,v_9$ be the probabilities of each of the nine words being picked as the changed one, $\sum_{i=1}^9v_i = 1$.

Now, we have to interpret what you mean by "due to chance alone". I suspect that it pertains to both the phase where the person answers "yes" or "no", and to the picking of the changed word. The only meaning I can attach to such an expression, is
A) "due to chance alone = pick yes or no "randomly" = yes and no have the same probability", so $q= \frac 12$.
B) "due to chance alone = picking a word "randomly" = all words have the same probability of being picked" so $c_1=...=c_9 = \frac 19$.
If A) and B) reflect what you mean, then, the answer to the question

"What is the probability of correctly answering that a word has changed, AND which word it was, due to chance alone" is :

$$P(\text{changed version}\land \text{yes} \land \text {correct word})= p\frac 12\frac 19 =p\frac {1}{18}$$

If now we want the probability conditional on the event that the person is shown a changed version, then we have

$$P( \text{yes} \land \text {correct word}|\text{changed version})= \frac 12\frac 19 =\frac {1}{18}$$

We arrived at the one of your numbers. Let's assume that we always show a changed version, i.e. $p=1$.

Now, if you change the design of your experiment, and the "yes/no" answer is not placed sequentially before the choice of the words, but the "no" answer is placed as a "tenth option" (as the other person suggested to you) along side the nine words, and all ten are equiprobable then the probability to answer "yes" and to pick the right word becomes, by standard probability rules $$P(\text{yes} \land \text {correct word}) = P(\text {yes} |\text {correct word} ) \cdot P(\text{correct word})$$ The probability to say "yes" given that you have picked the correct word is unity, since in order to have picked the right word you must have picked one of the choices that say some word. The probability of picking the correct word is $\frac {1}{10}$ so $$P(\text{yes} \land \text {correct word}) = 1\cdot \frac {1}{10} = \frac {1}{10}$$ ...which is the second of your numbers. So you are both right, only, you are talking about different choice (and therefore probabilistic) set ups.

i believe the answer is actually neither 1/18 nor 1/10 :)

take one of those old high school scantron quizzes/tests instead of a reading test.. remember the multiple-choice questions that you'd fill in with a #2 pencil?

question #1 of the multiple choice quiz has two possible answers. if you flipped a coin to answer question #1, you'd have a 50% chance of guessing correctly. so people who don't know the answer and blindly guess will average 50% on this question.

question #2 has nine possible answers. so among people who attempt question #2 but don't know the answer to question #2 and blindly guess..they'll average 11.111% correct on this question. but not everyone will get here! two groups of your test-takers will have the opportunity to get here: 100% of the test-takers who knew the correct answer to question #1 and 50% of the test-takers who guessed at random.

here's a critical piece of information: what share of your test-takers will know the correct answer to question one but randomly guess on question #2?

if 100% of your test-takers don't know the answer to either question, then i think the probability of answering both questions correctly is 1 out of 18. but a certain proportion of your test-takers will know the answer to your first question but not your second question..and they're going to throw off this probability calculation.

• +1 This helps drive home the point that the question needs additional information to be solvable and it shows just what kind of information is needed.
– whuber
Aug 26, 2013 at 15:20