1
$\begingroup$

Long story short: there used to be a TV show that always ended with 2 contestants having to answer 5 questions each (alternating); at some point I stopped watching it because it looked so blatantly rigged to me, but I would like to know if statistics can tell if my impression was right or not.

Basically, there were incredibly easy questions, like 'what European country has Paris as its capital?', and incredibly difficult ones, like 'what was the name of the actor who played Antoine Doinel in a famous 1959 French movie?', with very little in between.
This would not be a (big) problem in itself, except for the fact that quite astoundingly, occasionally there were long series of episodes where one particular contestant always got at least 4 or even all 5 fairly easy questions and kept winning evening after evening, whereas the other one always got 4 or 5 very difficult ones (and lost).
At some point this became so absurdly biased that, based on the first question that was asked of each contestant, you could already tell who was going to win, with something like 90% accuracy. And it seemed to me that often the 'favoured' contestant was one who had won a lot of money the previous evening, as if they wanted to 'keep' him/her, because the public got 'hooked'... Some contestants ended up 'staying' for 10-15 evenings, which is really a lot, considering that the average 'lifetime' was about 1-2 evenings per contestant.

Coming back to maths, I tried to find a way to calculate how likely it was to have 5 easy questions on one side and 5 difficult ones on the other side under the hypothesis that, as they claimed, the questions were randomly chosen by a 'computer'.
Imagine that the set of all questions was very large and that 50% were the difficult ones (let's call them $D$), and 50% the easy ones ($E$). So $p=p_E=p_D=1/2$.

Normally, the binomial distribution would predict that 50% easy questions and 50% difficult ones is the most likely outcome ($\approx 25 \%$). But that's when it doesn't matter in what order the questions are picked, I think, as in the formula you have indeed that binomial factor that multiplies the probability of one 'sequence' of D and E by all possible ways in which that number of D's and E's could occur.

What if the order does matter? If I'm not mistaken, in that case you just use the probability of the sequence itself, omitting the binomial factor.
Then if the first 5 questions are for contestant 'A' (the supposedly favoured one) and the next 5 for contestant 'B', the sequence $EEEEE \ DDDDD$ has probability $p_E^{5} \cdot p_D^{5} = p^{10} \approx 0.1 \%$.

And that's where I am stuck.

I think I can't draw any conclusions from a single observation of this kind of outcome (Paul the octopus and all that). Is that so? Or does the fact that the probability is so low make this kind of event already 'suspicious'? But then, a sequence like $EDDEE \ DEDDE$ has exactly the same probability, and I would not find it suspicious... So perhaps the kind of probability I am calculating is wrong. Maybe a 'runs test' would apply instead?

Then, suppose this sequence $EEEEE \ DDDDD$ is observed in 5, 6, ..., 10 consecutive episodes (which was actually the case). What is the correct way to test the hypothesis that the questions were not chosen at random in those cases?

Any suggestions on what reading I can do / where I should look for an answer to this?

Thanks!


EDIT - after reply by gunes

I worked a bit more on this, starting from a simpler example.

Suppose you have to ask 3 questions, taken at random from the usual 50:50 set.

Then the possible sequences (and between brackets, the number of separated 'groups' of question types in each) are:

$$EEE (1), EED (2), EDE (3), DEE (2), EDD (2), DED (3), DDE (2), DDD (1)$$

So, summarising the number of sequences by number of groups:

1 group: 2
2 groups: 4
3 groups: 2

I did the same for 4 questions. Summary:

1 group: 2
2 groups: 6
3 groups: 6
4 groups: 2

I hope it's not a coincidence, but these look to me like the classic binomial terms multiplied by 2, $2 \cdot (1,2,1)$ and $2 \cdot (1,3,3,1)$.

If that is the case and the concept can be extended to $N$ questions, then the number of sequences with $g$ separated groups of question types should be:

$$2 \cdot \binom {N-1} {g-1}$$

E.g. for $N=4$ and $g=2$:

$$2 \cdot \binom {4-1} {2-1} = 2 \cdot \binom {3} {1} = 6$$

The total number of possible sequences is:

$$\sum_{i=0}^{N-1} {2 \cdot \binom {N-1} {i} } = 2^N$$

Therefore the probability of a sequence belonging to the '$g$ groups' set is:

$$P(N,g) = \frac {\binom {N-1} {g-1}} {2^{N-1}}$$

So coming back to my initial question, it seems that the probability of a sequence with 2 groups in 10 questions is:

$$P(10,2) = \frac {\binom {10-1} {2-1}} {2^{10-1}} = \frac 9 {512} \approx 0.0176$$

But I'm not sure this is what I want, because it puts together sequences like $EEEEE \ DDDDD$ and $EDDDD \ DDDDD$ or $EEEEE \ EEEED$, which would not raise so much suspicion perhaps.

Am I still barking up the wrong tree?


EDIT 2 - after more thought

I checked this theory in a basic statistics book. They make the example of tossing 3 coins.
The possible sequences are the same as I exemplified before, substituting $E=heads$ and $D=tails$.
However, at that point they count the number of heads in each sequence, and show that: $$HHH (3), HHT (2), HTH (2), THH (2), HTT (1), THT (1), TTH (1), TTT (0)$$

Thus:

0 heads : 1
1 head : 3
2 heads : 3
4 heads : 1

But what if I counted the sequences that have at least two consecutive heads? Then:

at least one HH : 3
no HT : 5

It looks like, depending on how you decide to interpret or group the various outcomes, you have different distributions.

So my doubt remains: confronted with a specific sequence, what can we infer about the way it was obtained?

There's an example in the book 'Naked Statistics' where the lecturer asks a large audience of students to toss a coin several times. As soon as someone gets a tail, they must stop, otherwise they continue. Apparently it often happens that someone manages to 'stay in the game' when most of the other people have stopped.
His point, I think, is that even 'rare' events, like getting a long sequence of heads, can be observed if you repeat the experiment a large number of times.
Trouble is that sometimes we only get one shot at observing an event. When Paul the octopus correctly guessed the future results of several football matches in sequence, that was it, it was not done again to see if that event was a random aberration. If based on that event I rejected the hypothesis that the octopus was unable to predict football match results, I'd run a very small (but not zero) risk of being wrong, and perhaps in that case I would indeed be wrong.

So OK, the event '100 heads' has the same probability as '50 heads followed by 50 tails', or any other individual sequence in fact; but when we interpret the sequences by their composition in terms of heads and tails rather than by their exact order, we can say that more 'disordered' sequences that can be made in several different ways are more frequent, and observing them casts less doubt on the null hypothesis.

I think my error here was to neglect the fact that I was looking at the two contestants separately, and I should have done that when considering the outcomes.
The 'favoured' contestant getting a sequence of 5 easy questions is an event with probability $1/32$. Same for the 'disfavoured' contestant getting a sequence of 5 difficult questions. The probability that these two events happen simultaneously is the product of the two, i.e. the probability I calculated initially, $1/2^{10}$.
If I observed a sequence with the overall same number of $E$ and $D$, but distributed differently between the two contestants, like:

$$EDDEE \ DDEDE$$

now the probability of each contestant's sequence regardless of the order within each is much higher, and in particular equal to $p^5 \cdot \binom {5} {2} = 5/16$, and the probability of observing this situation is $(5/16)^2 \approx 10 \%$.

So in general I think that the probability of a particular outcome defined as having $n_A$ easy questions for contestant 'A' and $n_B$ easy questions for contestant 'B' within the above hypothesis is:

$$p^{10} \cdot \binom {5} {n_A} \cdot \binom {5} {n_B}$$

I checked numerically: the sum of these probabilities is 1 when all possibilities are considered (0,0),(1,0),...,(5,5).

So I don't know, maybe I'm rambling, but this at the moment seems to make sense to me.
If not, suggestions are welcome!

$\endgroup$
  • $\begingroup$ If there are 50% easy questions and 50% hard questions in a huge pool and the computer picks five at random, the probability of getting $X$ difficult questions out of $n = 5$ is $\mathsf{Binom}(n=5, p=1/2).$ The probabilities $P(X=i)$ are $1/32, 5/32, 10/32, 10/32, 5/32, 1/32,$ for $i = 0, 1, \dots, 5,$ respectively. $\endgroup$ – BruceET Sep 11 '18 at 3:27
1
$\begingroup$

This is deeply related to Typical Sets in Information Theory. In your case, the $EEEEE$ $DDDDD$ is not a member of the typical set. Another example would be tossing a fair coin 100 times, and having all Heads. Its probability is the same with any other sequence, but it'd surprise you.

$\endgroup$
  • $\begingroup$ Thank you gunes. I looked up the article you suggested. I sort of get that it is related to my case here, but I don't see any immediate way to use the formulae mentioned there to calculate the probability I need. I tried something else in the edit of my question. $\endgroup$ – user6376297 Sep 11 '18 at 19:27
  • $\begingroup$ BTW, this seems also related to a deeper doubt I've always had and never really figured out. You probably know that people in some countries play a game where they have to guess some numbers that are extracted (without replacement) from a set of integers between 1 and 90. Occasionally it happens that a specific number is 'late', i.e. it does not get extracted for several months. Well, people believe that this makes it 'more likely' to be extracted next time. But in principle that is not true. How to reconcile the randomness of the 'next extraction' with the 'typical sequence' story? $\endgroup$ – user6376297 Sep 11 '18 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.