2
$\begingroup$

Edit: Revising original question

I'm trying to make sure I understand frequentist confidence intervals. Below is a hypothetical setup that illustrates the way I'm trying to think about this, and I'm wondering where/if I'm going wrong?

Setup: A computer generates 100 numbers from a binomial distribution, B(100,p) with the parameter p only known to the computer. The computer also calculates a clopper-pearson 95% confidence interval along with the data. There is an exchange on which the following contract can be bought and sold: "This contract pays out \$1 if the confidence interval contains p." At the end of the week, the computer reveals p to all market participants so that the contract can resolve.

My question relates to the fair value of the contract. BEFORE the numbers are generated, clearly the contract has a fair value of \$0.95. AFTER the numbers and interval are generated, unless there was something obviously absurd about the confidence interval, wouldn't the fair value still be roughly be \$0.95? So that in the presence of uncertainty, it's often reasonable to think about a 95% confidence interval as having a 95% chance of containing the parameter? And I've seen Jaynes in his book give examples where the confidence interval is obviously wrong, but most of the time that doesn't seem to be the case (if there was not much uncertainty, wouldn't there be no role for statistics?)

The "fair value" of the contract in this setup is how I am thinking about probability and is a proxy for the "probability of containing the parameter."

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ This is a fine question, but you should try to think of a less generic title. $\endgroup$
    – Matthew Drury
    Commented Aug 28, 2016 at 21:16
  • 1
    $\begingroup$ Please summarize whatever materials are necessary to understand the question -- people will be more inclined to answer if your question is self-contained. $\endgroup$
    – Sycorax ♦
    Commented Aug 28, 2016 at 21:42

2 Answers 2

Reset to default
2
$\begingroup$

Found an exchange on andrew gelman's blog that is exactly what I was trying to ask:

Thinking a lot more about this, it seems like Fisher's fiducial inference is similar to what I was trying to get at, although it seems like an unaccepted/unclear theory https://en.wikipedia.org/wiki/Fiducial_inference

John says:

March 15, 2014 at 2:50 pm

I agree Andrew. I never teach my students to associate the word confident with the interval and try to describe it as just a way to label the interval. It could be the “orange” interval but the label we’re using is descriptive of the method.

However, if you genuinely are not in a situation where you can have any further certainty about whether the interval does, or does not contain the true value, then you can know the method you used makes you correct about the interval containing the mean 95% of the time. Some might call that 95% confidence. From a Bayesian perspective you might argue that’s a rare occurrence, or that it never occurs. But that’s a separate philosophical debate. I think that your average undergrad doing a project where they estimate an interval on a fairly large effect probably has pretty good standing to claim 95% confidence, whereas a scientist who estimates an interval containing 0 where there are sound reasons it should not be in the interval can probably exclude some of the range and 95% doesn’t apply post hoc. Perhaps if one is clear that they have the confidence prior to the appearance of the interval and not afterwards things would be ok.

So while I agree that you that the statement is not strictly correct in a general case and I further agree that it should not be taught as a way of stating the CI. I’m not sure I agree that it’s fairly tested in the given experiment.

Andrew says:

March 15, 2014 at 2:58 pm

John:

Yes, I agree that it depends on context. In lots of simple problems, the “we can be 95% confident” formulation is just fine. I was pushing back against Russ’s statement that classical confidence intervals are “the very meaning of ‘confidence’.” But in many cases, all definitions will approximately agree.

But to get back to my point in the above post, I’d argue that we use flat priors (or the equivalent classical procedures) in all sorts of problems where they’re not appropriate. My own thinking on this has indeed changed a lot in recent years, and my latest views have not even fully made their way into the latest version of Bayesian Data Analysis.

Edit (deleted example I had below, didn't think it was good)

Improve this answer
$\endgroup$
1
$\begingroup$

I think Tarpey's logic and yours go astray by confusing frequentist and Bayesian notions of probability. Under frequentist conceptions, which are the context of confidence intervals, probability isn't epistemic; it's the limiting frequency of a random process. The confidence of a 95% confidence-interval construction procedure (that is, a lower bound for the probability that intervals constructed by the procedure will contain the true value for which they are constructed) is 95% regardless of who you are, what you know, and how much data has been collected. Importantly, the probability belongs to the procedure that constructs the confidence interval, rather than any individual interval.

If you want an interval that really does have a 95% probability of containing a population value, and you're ready to adopt the Bayesian, epistemic view of probability that's necessary for this, you want not a confidence interval but a credible interval. Importantly, a 95% confidence interval need not also be a 95% credible interval.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your answer. I'm trying to think about this from a "common sense" point of view instead of having to ascribe to a certain "notion/epistemic view". In the example I gave in the OP, suppose I was given the option to buy a contract that pays me \$1 if the 95% confidence interval contains the true value. BEFORE seeing the data, I think we could agree that the fair value of the contract is \$0.95. AFTER the data is revealed, excluding cases in which me seeing the data gives me more information about the unknown parameters, wouldn't the contract still be reasonably valued at \$0.95? $\endgroup$
    – convolutedstatistic
    Commented Aug 30, 2016 at 14:03
  • $\begingroup$ @convolutedstatistic "I'm trying to think about this from a 'common sense' point of view instead of having to ascribe to a certain 'notion/epistemic view'." — You need a coherent way to interpret probabilities in order to use them for anything. $\endgroup$
    – Kodiologist
    Commented Aug 30, 2016 at 14:25
  • $\begingroup$ But can't I both have quantified beliefs that satisfy kolmogorov's axioms AND also think about probability in terms of repeated sampling? $\endgroup$
    – convolutedstatistic
    Commented Aug 30, 2016 at 16:00
  • $\begingroup$ @convolutedstatistic No. Probability satisfies the Kolmogorov axioms whether interpreted in a frequentist or Bayesian fashion, but the degrees-of-belief (Bayesian) and limiting-frequency (frequentist) views of probability are distinct, hence, e.g., the non-equality of confidence intervals and credible intervals. $\endgroup$
    – Kodiologist
    Commented Aug 30, 2016 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.