Calculating the probability of being correct due to chance

Question

I'm looking for a third opinion on the probability of making a particular decision due to chance alone.

I have a task where people read a paragraph of text, then they are shown it a second time, except some of those times one word has changed. People are then asked whether they detected a change. If they say no then that's the end of the task. If they say yes, then they are given a set of nine words and are asked to respond which word was changed.

If people correctly guess that a change has occurred AND which word was changed, then a single "correct" is registered. If they correctly detect a change and wrongly guess the word that has changed then an "incorrect" is registered (similarly, it is incorrect if a word has changed but the person did not detect it).

What I'm trying to work out is what is the probability of correctly answering that a word has change, AND which word it was, due to chance alone. My first thought was that it was 1/18. That's a 1/2 chance of firstly correctly guessing that a change had occurred, followed by a 1/9 chance of correctly guessing the word.

Someone I've spoken to has suggested that the chance rate is actually 1/10. In effect, "no, a change has not occurred" becomes the tenth option.

So, is the chance of being correct 1/18, 1/10, or are we both wrong?

Answer 1

Let $p$ be the probability that we show a changed version. Let $q$ be the probability that the person says "yes there is a change". Let $v_1,...,v_9$ be the probabilities of each of the nine words being picked as the changed one, $\sum_{i=1}^9v_i = 1$.

Now, we have to interpret what you mean by "due to chance alone". I suspect that it pertains to both the phase where the person answers "yes" or "no", and to the picking of the changed word. The only meaning I can attach to such an expression, is
A) "due to chance alone = pick yes or no "randomly" = yes and no have the same probability", so $q= \frac 12$.
B) "due to chance alone = picking a word "randomly" = all words have the same probability of being picked" so $c_1=...=c_9 = \frac 19$.
If A) and B) reflect what you mean, then, the answer to the question

"What is the probability of correctly answering that a word has changed, AND which word it was, due to chance alone" is :

$$P(\text{changed version}\land \text{yes} \land \text {correct word})= p\frac 12\frac 19 =p\frac {1}{18}$$

If now we want the probability conditional on the event that the person is shown a changed version, then we have

$$P( \text{yes} \land \text {correct word}|\text{changed version})= \frac 12\frac 19 =\frac {1}{18}$$

We arrived at the one of your numbers. Let's assume that we always show a changed version, i.e. $p=1$.

Now, if you change the design of your experiment, and the "yes/no" answer is not placed sequentially before the choice of the words, but the "no" answer is placed as a "tenth option" (as the other person suggested to you) along side the nine words, and all ten are equiprobable then the probability to answer "yes" and to pick the right word becomes, by standard probability rules $$P(\text{yes} \land \text {correct word}) = P(\text {yes} |\text {correct word} ) \cdot P(\text{correct word}) $$ The probability to say "yes" given that you have picked the correct word is unity, since in order to have picked the right word you must have picked one of the choices that say some word. The probability of picking the correct word is $\frac {1}{10}$ so $$P(\text{yes} \land \text {correct word}) = 1\cdot \frac {1}{10} = \frac {1}{10} $$ ...which is the second of your numbers. So you are both right, only, you are talking about different choice (and therefore probabilistic) set ups.

Answer 2

i believe the answer is actually neither 1/18 nor 1/10 :)

take one of those old high school scantron quizzes/tests instead of a reading test.. remember the multiple-choice questions that you'd fill in with a #2 pencil?

question #1 of the multiple choice quiz has two possible answers. if you flipped a coin to answer question #1, you'd have a 50% chance of guessing correctly. so people who don't know the answer and blindly guess will average 50% on this question.

question #2 has nine possible answers. so among people who attempt question #2 but don't know the answer to question #2 and blindly guess..they'll average 11.111% correct on this question. but not everyone will get here! two groups of your test-takers will have the opportunity to get here: 100% of the test-takers who knew the correct answer to question #1 and 50% of the test-takers who guessed at random.

here's a critical piece of information: what share of your test-takers will know the correct answer to question one but randomly guess on question #2?

if 100% of your test-takers don't know the answer to either question, then i think the probability of answering both questions correctly is 1 out of 18. but a certain proportion of your test-takers will know the answer to your first question but not your second question..and they're going to throw off this probability calculation.