We know that someone identified correctly 3 out of 5 of wines he tasted. How can we answer if he can do that consistently with randomisation testing? [closed]

Question

I don't know what should i choose as a control function for that problem. Thanks for your time.

Answer 1

Dealing with wines introduces complexities that I do not want to address. For instance, if I guess that the wine is from 2019 but it is from 2018, but I get the vineyard, etc correct, did I guess right or wrong or just slightly wrong? However, a more straightforward example is a magician guessing cards.

A magician might claim that you can shuffle a deck of $52$ cards (standard deck), draw a card, and have him guess the card. I propose a strategy for guessing: always guess the two of clubs. I should get the right answer about $2\%$ of the time, since $1/52\approx 0.02$.

Therefore, assuming the card goes back in the deck for a new trick, the baseline rate is that $1/52$. In order to be better than guessing at random, you need to do better than $1/52$.

Maybe the magician misses his guesses sometimes, but in $5$ guesses of the card, the magician guesses the right card $3$ times, much like your wine taster. That's a $60\%$ success rate, quite a bit higher than the $\sim 2\%$ success rate from random guessing.

Let's test if this is statistically significantly better than $1/52$.

prop.test(3, 5, 1/52)

This returns a tiny p-value. That is, it would be extremely unlikely for someone who is only as good at randomly guessing the card to be able to guess three cards in five attempts.

If you can get a baseline rate of how often one should be able to guess the correct wine by guessing at random (probably 1/(# of different wines)), I do not believe that you need any kind of control for your wine problem.