3
$\begingroup$

There are numerous threads on CV about the definition of and intuition about a confidence interval. However, I was surprised that none contained my intuitive reasoning about a confidence interval, so perhaps it is wrong.

Here it goes:

The confidence interval defines a range of hypothetical sample effects which — under the assumption that the observed sample effect is identical to the true effect — would not be surprising (where the criterion surprising/not surprising is a function of the confidence level and is identical to the interval bounds).

Is there anything wrong about this statement?

(NB: if it is a correct statement, I'd still understand why it is not the common explanation, because it's hard to bring the meaning of the exact confidence level (e.g. 95%) into this.)

(ps: since many answers in other threads are hard to comprehend for me, it is possible, that my reasoning was expressed elsewhere.)

$\endgroup$
18
  • 1
    $\begingroup$ I feel like this is exactly how people usually describe confidence interval intuition and interpretation. Do you mean to ask why this isn’t a mathematical definition of a confidence interval? (In that case, couldn’t you basically say the same about a credible interval?) $\endgroup$
    – Dave
    Commented Sep 13, 2023 at 13:12
  • 2
    $\begingroup$ A mathematical definition is not the same as an intuition about interpretation. You seem to have linked four discussions of the former and now ask about the latter. $\endgroup$
    – Dave
    Commented Sep 13, 2023 at 13:26
  • 4
    $\begingroup$ Many might find this characterization of CIs problematic because it relies on undefined and unusual terms like "hypothetical sample effect" as well as on vague qualitative characterizations like "surprising." It isn't clear enough even to be wrong. $\endgroup$
    – whuber
    Commented Sep 13, 2023 at 14:33
  • 1
    $\begingroup$ I don't think your assumption "the assumption that the unknown true effect corresponds to our observed sample effect " is clear. Please update the question to make this assumption understandable. (What does "correspond" mean?) $\endgroup$ Commented Sep 13, 2023 at 18:37
  • 1
    $\begingroup$ I suspect the phrase "hypothetical sample effect" would be understood correctly only by somebody who already knows what a CI is and is capable of making the translation. The second part of your phrase, "the assumption that the observed sample effect is identical to the true effect," is not part of any CI definition or concept. $\endgroup$
    – whuber
    Commented Sep 13, 2023 at 21:53

2 Answers 2

1
$\begingroup$

Confidence intervals can be interpreted in two ways and it depends on whether you use a double negative or not.

A confidence intervals contains those parameter values:

  • For which the observation is not unlikely
  • For which the observation is likely

In your definition you take the first approach.

The second interpretation makes it more like an interval based on a fiducial distribution, which expresses how different values are more or less plausible.


The two approaches, are not describing a different interval. They are just a different viewpoint of the same thing. They stress different aspects of the interval and relate to the gray area in defining likely and unlikely.

Inference aims to find some optimal value (e.g. most likely) along with a range that expresses the uncertainty about the inference process. That range can be expressed in terms of values that are also likely or values that are not unlikely.

How it is called/considered depends on the setting and use of language. A value that is not unlikely is not neccesarily a value that is likely. For example imagine a scientific research that uses a cautious 99.999% confidence interval thay may contain a lot of values, many of them are not neccesarily likely and instead are just not unlikely according to some very strict 0.001% level for being unlikely.


it's hard to bring the meaning of the exact confidence level (e.g. 95%) into this

A mathematical definition that is a translation of your interpretation could be

$$CI(95\%) = \{ \theta: H(\theta;100\%-95\%) = false\}$$

In words: the 95% confidence interval/region is the range of parameter values $\theta$ for which a hypothesis test at a significance level of 5% fails. (a failed test could be regarded as the observation is not unlikely for that given parameter value, the value can not be rejected)


Just to clarify, as you characterize my explanation of a CI as containing "those parameter values for which the observation is not unlikely". Is this really the same as my statement, which essentially says that CIs contain "those parameter values of hypothetical samples which are not unlikely given the observed parameter value"?

No this is not the same.

The reason that I mischaracterized your definition is because I took a liberal approach while reading your definition and that relates to the points mentioned in the other answer and comment. For example, 'parameter' would be better than 'hypothetical sample effect' (I do not know what a 'sample effect' means, does a sample have an effect?).

I see now that I have passed beyond a certain important difference. You express something as 'not unlikely parameter values given the observation' and I translated it as 'parameter values for which the observation is not unlikely'.

There is a big jump to be made between the two. It relates a bit to the term 'inverse probability' (What exactly does the term "inverse probability" mean?). When we speak about probability and likely values, then we can understand this very well in the direction when the parameters are given and we predict the outcomes. In the other direction, when we know the outcomes, but do not know the parameters, then it is more difficult to speak about 'probability'. Terms like likelihood, confidence and fiduciality are used to replace the probability.

In your definition you use 'likely' applied to the inverse probability. That makes it difficult to interpret.

(I use the term as well, but it is in the other direction, the probability of the observation given the parameters. With my post it is however still a problem what I exactly mean with 'likely' and it is a common criticism of p-values which relates to the probability of the observed event or a more extreme event, while 'extreme' is not well defined.)

If you really want to stick to

those parameter values of hypothetical samples which are not unlikely

, then it is a wrong definition of the confidence interval.

This interpretation of the confidence interval might be close to what is actually a likelihood interval, but the likelihood interval and confidence interval are different. The confidence interval does not neccesarily contain the values with the highest likelihood. See also The basic logic of constructing a confidence interval. In the figure below you can see how the confidence interval boundaries relates to likelihood values that are not the same level (see the panel on the right where the red and green dots, depicting the boundaries, are not at the same likelihood value).

example

Legend: The red line is the upper boundary for the confidence interval and the green line is the lower boundary for the confidence interval. The confidence interval is drawn for $\pm 1 \sigma$ (approximately 68.3%). The thick black lines are the pdf (2 times) and likelihood function that cross in the points $(\theta,\hat\theta)=(-3,-1)$ and $(\theta,\hat\theta)=(0,-1)$.

$\endgroup$
14
  • $\begingroup$ Thanks a lot! Just to clarify, as you characterize my explanation of a CI as containing "those parameter values for which the observation is not unlikely". Is this really the same as my statement, which essentially says that CIs contain "those parameter values of hypothetical samples which are not unlikely given the observed parameter value"? $\endgroup$
    – monade
    Commented Sep 14, 2023 at 7:53
  • $\begingroup$ @monade I took a liberal approach while reading your definition and that relates to the points mentioned in the other answer and comment. For example, 'parameter' would be better than 'hypothetical sample effect' (I do not know what a 'sample effect' means, does a sample have an effect?). I see now that I have passed beyond a certain important difference. You express something as 'not unlikely parameter values given the observation' and I translated it as 'parameter values for which the observation is not unlikely'. There is a big jump to be made between the two.... $\endgroup$ Commented Sep 14, 2023 at 9:02
  • $\begingroup$ ... it relates a bit to the term 'inverse probability' (What exactly does the term "inverse probability" mean?). When we speak about probability and likely values, then we can understand this very well in the direction when the parameters are given and we predict the outcomes. In the other direction, when we know the outcomes, but do not know the parameters, then it is more difficult to speak about 'probability'. Terms like likelihood, confidence and fiduciality are used to replace the probability.... $\endgroup$ Commented Sep 14, 2023 at 9:07
  • $\begingroup$ ... in your definition you use 'likely' applied to the inverse probability. That makes it difficult to interpret. (I used the term as well, but it is in the other direction, the probability of the observation given the parameters. With my post it is however still a problem what I exactly mean with 'likely' and it is a common criticism of p-values which relates to the probability of the observed event or a more extreme event, while 'extreme' is not well defined.) $\endgroup$ Commented Sep 14, 2023 at 9:09
  • 1
    $\begingroup$ @monade An interval that relates to potential values/statistics of a new sample is a prediction interval. Confidence intervals relate instead to the value of a parameter that describes the population. $\endgroup$ Commented Sep 14, 2023 at 12:59
3
$\begingroup$

Firstly, good for you for attempting to come up with a new intuitive description of a statistical object. I'm going to give a critique of what you've come up with, but please don't let this put you off your attempt to refine this description, or to try to come up with your own descriptions of other statistical concepts. With that out of the way, here are a number of problems with your explanation:

  • Firstly, you refer to the CI as defining a range of "hypothetical sample effects", which does not generally match what you are making an inference about in a CI. In a typical CI you have some observable data and an unknown "parameter" and your CI gives a range of values of the latter. The parameter that is the object of inference could be pretty much any unobservable aspect of the process --- it does not necessarily correspond to a "sample effect". Consequently, describing the object of inference using the term "hypothetical sample effect" is vague at best, and probably just incorrect in a range of problems.

  • Secondly, your reference to the assumption that "the observed sample effect is identical to the true effect" is unclear. Again, this seems vague at best and wrong at worst. What is the "sample effect" here? What is the "true effect" here? The true effect of what? (Is this some veiled reference to an unknown "parameter"?) Does this formulation assume some causal mechanism, and if so, how is it applicable to statistical inference problems that look only at predictive inference?

  • Finally, your description of the substantive requirement for the confidence interval procedure does not really give any clear description of (or even a reasonable allusion to) the actual mathematical criterion at issue. You merely say that the derived range of values "would not be surprising" and that this notion of surprise "is a function of the confidence level and is identical to the interval bounds". (I have no idea what you mean by asserting that the concept of surprise "is identical to the interval bounds" --- that is clearly wrong, so set that part aside.) At best, this lets me know that the procedure gives a range that is not "surprising", where the latter is determined by the confidence level. Effectively, this tells me that the confidence level affects the interval somehow, but how is not specified even vaguely. That is minimal information --- you have not told me what the confidence level actually measures or anything at all about how it affects the interval.

The outcome of these problems is this: you refer to objects and effects that are vague at best, and probably inapplicable at worst, and you tell me nothing about the actual substantive requirement for forming a confidence issue. Personally, I do not find any value in this particular description (though again, I commend you on your attempt).

$\endgroup$
6
  • $\begingroup$ Hi Ben, thanks a lot for your thoughtful, constructive and encouraging response, much appreciated. I'm not sure which format would be best to respond to this, but I'll try in the comments (please let me know if there's a more appropriate way). $\endgroup$
    – monade
    Commented Sep 14, 2023 at 7:07
  • $\begingroup$ Re first point: this is probably sticking my neck out a bit too far, but I'd argue that your statement that CIs give a range of values for the unknown parameter is precisely why it is often mistakenly believed that the CI gives the probability for the parameter to be contained in the CI. Since the CI is based on the sampling distribution (and constructed around the mean of the sampling distribution), is it not more correct to say that the CI gives a range of values for hypothetical sample effects and not for the unknown parameter? $\endgroup$
    – monade
    Commented Sep 14, 2023 at 7:07
  • $\begingroup$ Re second point: as a non-statistician I don't really see the issue here, sorry. By true effect I simply mean the true value of the parameter (or the "true effect") in the population. Perhaps this confusion is caused by my use of "effect", so please replace "true effect" with "true parameter" and "sample effect" with "this parameter computed for a sample" where relevant. $\endgroup$
    – monade
    Commented Sep 14, 2023 at 7:08
  • $\begingroup$ Re third point: again, excuse my ignorance, but I don't see how the use of the word surprising seems so unclear (not only to you, but also the commentators, so this is clearly on me). On the one hand, whether a hypothetical sample effect is considered surprising in my explanation is precisely defined in terms of whether hypothetical sample effects are inside or outside the bounds. And to me it also seems intuitive to say that a hypothetical sample effect would be considered surprising if it is far away from the actual observed sample effect (so far away that it is outside the CI bounds). $\endgroup$
    – monade
    Commented Sep 14, 2023 at 7:08
  • $\begingroup$ Now to your final point that my explanation merely tells you "that the confidence level affects the interval somehow, but how is not specified even vaguely": I entirely agree. This is what I referred to when I said "I'd still understand why it is not the common explanation, because it's hard to bring the meaning of the exact confidence level (e.g. 95%) into this.". $\endgroup$
    – monade
    Commented Sep 14, 2023 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.