2
$\begingroup$

I'm rather new to statistics. I'm learning on my own and I'm trying to understand confidence levels. I read, "How Confident Do You Need to be in Your Research?" by Jeff Sauro, Ph.D., 5 Jan 2015, and in it he states:

if you find that there’s little or no downside to being wrong, and if you have to pick among poor alternatives, 51% confidence is at least better than flipping a coin.

This seems to imply that 50% confidence is equal to flipping a coin, but I disagree. My logic is that flipping a fair coin has 0% confidence it will land on heads, this means there's an equal chance it could land on heads, you have 0 conjecture WHICH it will be. A situation with 50% confidence would be like a "loaded"(??) coin that WILL land on heads 50% of the time. The other 50% of the time the coin has an equal chance of landing on heads or tails. I'm clearly incorrect in this thinking and must be confused about what confidence level measures, but am unable to grasp it.

How is a 50% confidence level like flipping a coin? Would it have something to do with confidence interval width?

$\endgroup$
1
  • 1
    $\begingroup$ When you equate the event "the confidence interval covers the parameter value" and "the coin lands heads," you will get close to the basis of this analogy. A technical issue is that a 50% confidence interval procedure doesn't have to have a 50% chance of covering the parameter in all cases--but in some textbook settings it does. $\endgroup$
    – whuber
    Jun 1, 2022 at 13:33

2 Answers 2

6
$\begingroup$

I also find the coin flipping relation a little unintuitive. Let's start instead by imagining a game where I roll a 100-sided die and you want to guess the number. But instead of just guessing the exact number, you're allowed to guess a range of numbers from 1-100.

For example, I roll the die and get a number $X$, but I don't tell you what the number is. Then, you decide to guess the range of numbers from 1-95. At this exact moment in time, before I tell you if you're right or not, how could we describe the knowledge you have and how likely you are to be correct? There's a good chance that you've guessed the right number, you guessed 95 out of the possible 100 values. But technically we can't say "the probability you're correct is 95%". Because, I already rolled the dice and I know whether your guess is correct or not. There isn't anything random left to do, the result is already fixed. Instead we would say that you are 95% "confident" that the number I rolled is in the range you guessed.

Back to the casino, say that we decide to gamble on the outcome of this dice game. If the number from the die is in your range I'll give you \$10, if it's not you give me \$10. But to make it fair you're only allowed to guess a range of 50 numbers, not as many as you want. One round of the game might be that I roll the die and you guess the numbers 1-50. Then you are "50% confident" that your interval contains the number I rolled.

How does this game compare to gambling on a coin flip? You might be able to tell that this game I just described and gambling on the flip of a coin are statistically equal. The 100-sided die game is a little more complex, but expected results from both games are the same. You should win each game about 50% of the time.

So, you being 50% confident in the die game interval is equivalent to betting on a coin flip. If instead I allowed you to guess a range of 51 numbers in the die game, you would prefer that to a coin flip game because you're more likely to win. You shouldn't bet your house on it, but if we're playing for smaller amounts of money you'd probably like to take that 51% chance multiple times.

$\endgroup$
3
  • $\begingroup$ What would a 0% confidence mean in this example? $\endgroup$ Jun 1, 2022 at 12:45
  • $\begingroup$ @MelanieShebel If you guess the number is between 101-200, confidence in your guess would ideally be 0% $\endgroup$
    – Firebug
    Jun 1, 2022 at 13:35
  • 1
    $\begingroup$ @MelanieShebel 0% confidence means that you are certain your guess is wrong. In practice this shouldn't happen. But some examples could be guessing a number outside the range as Firebug suggested or guessing the result of a single 6-sided die is 3.5. A less intuitive example would be guessing the height of a tree is exactly Pi meters tall, with exact precision. $\endgroup$ Jun 2, 2022 at 4:49
0
$\begingroup$

The context is this phrase

I’m often asked what the best level of confidence to use is. The answer is that it depends on the consequences of being wrong.

For any application where an estimation with a confidence interval is involved you need to determine a convenient/practical desired level of confidence to compute intervals.

To determine this level you need to consider the advantages and disadvantages of making a wrongly estimated interval that does not contain the true parameter.

If you use a 50% confidence interval then this is like flipping a coin. Half the time you will be wrong and half the time you will be right (or depending on the definition of the confidence interval, at least half the time you are right). If you have a business where the cost of a being wrong is equal to the benefit of being right, then a 51% confidence interval can make you money in the long run in a similar way as the small house advantage of a casino.

It is an extreme example to show how the desired confidence level can vary based on the application. The proper level depends on the costs and benefits.


It is a bit black-and-white thinking. The costs and benefits are often not so discrete. Possibly in the case of betting it is. E.g. if you are betting on horse races then you either lose or win. But in some business situation you can have a varying degree of costs and benefits depending on how far away from the actual true value you are. Say you estimate that some value is between 2 and 6 with 90% confidence and you act according to this, then the consequences might be more or less dramatic depending on whether the true value is 6.1 or 10.

The confidence intervals are often a rule of thumb and a rough indication. The coin flipping example is an example how the levels can differ depending on the application, but you are not often gonna find those example applications in real-life.

$\endgroup$
7
  • 1
    $\begingroup$ Re "Half the time you will be wrong and half the time you will be right." Maybe. From the definitions all you can say generally is that at least half the time you will be right, and in some (perhaps almost all) conceivable cases you might be right more than half the time. Although the equation with 50% confidence and coin flipping can be perfectly accurate, it's important to describe the settings in which those are genuinely equivalent. $\endgroup$
    – whuber
    Jun 1, 2022 at 13:30
  • $\begingroup$ @whuber, you mean the definition of the $\alpha\%$ level confidence interval as an interval that contains the true parameter at least $\alpha\%$ of the time? In which case it some form of an 'approximate' confidence interval. $\endgroup$ Jun 1, 2022 at 13:33
  • $\begingroup$ Not quite. I am thinking particularly of (non-randomized) CIs for parameters of discrete distribution families. $\endgroup$
    – whuber
    Jun 1, 2022 at 13:45
  • $\begingroup$ But there are different definitions. Some speak about exact probabilities, others about approximate probabilities or inequalities. $\endgroup$ Jun 1, 2022 at 13:50
  • $\begingroup$ I have never encountered a definition of confidence interval that relies on "approximate probabilities." Do you have a reference in mind? $\endgroup$
    – whuber
    Jun 1, 2022 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.