5
$\begingroup$

Let there be a repeatable real world experiment with two outcomes denoted by $0,1$ for convenience (Tossing a coin for example). Let $X_i$ be the random variable that models the ith repetition of the experiment. It is an assumption of our model of that real world phenomenon that $X_1,X_2,X_3,...$ are independent identically distributed to $B(1,p)$. I noticed in all confidence intervals for $p$ I encountered so far, one basically throws away all information in the sample and only keeps track of the total number of 1(s) (Total number of heads in case of a coin).

I came up with the following confidence interval. First let me make clear my defintion of a confidence interval in case we are working with a sample of size $n$.

Defintion: A $1-\alpha$ confidence interval for the parameter $p$ above are random variables $L,U$ that are functions of our sample $X_1,X_2,...,X_n$ such that $P(L<p<U)\geq 1-\alpha$. It is almost similar to the defintion of my book.

Using this defintion, I proceed to design my own confidence interval. Suppose our sample size is even of size $n=2k$. Set $\overline{X}$ to be the average of $X_1,X_2,X_3,...,X_{2k}$, and set $\overline{Y}$ to be the average of $X_2,X_4,X_6,...,X_{2k}$ By Chebyshev inequality, we have the inequalities below: $$P(\overline{X}-\frac{1}{2\sqrt{k\alpha}}<p<\overline{X}+\frac{1}{2\sqrt{k\alpha}})\geq 1-\frac{\alpha}{2}$$ $$P(\overline{Y}-\frac{1}{\sqrt{2k\alpha}}<p<\overline{Y}+\frac{1}{\sqrt{2k\alpha}})\geq 1-\frac{\alpha}{2}$$ By Inclusion exclusion principle, we get: $$P(\overline{Y}-\frac{1}{\sqrt{2k\alpha}}<p<\overline{X}+\frac{1}{2\sqrt{k\alpha}})\geq 1-\alpha$$. Thus, we get a $(1-\alpha)$ confidence interval which is $]\overline{Y}-\frac{1}{\sqrt{2k\alpha}},\overline{X}+\frac{1}{2\sqrt{k\alpha}}[$ Ofourse one could even consider more interesting statistics (more interesting than $\overline{Y}$)from the sample like for example the number of occurrences of the strings $1,0,0,0,1$ in the data collected.

Question : Now suppose we use the above confidence interval with confidence 99% for the case of tossing a coin $2\times 10^{30}$ times and it happens that we get the sample realization $0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,.....$, then applying the confidence interval above that I designed will give approximately something like $0.999...<p<0.50001$. I am not sure how to interpret the result of this confidence interval in this case. Should the interpretation be that our model that $X_1,X_2,...$ independent identically distributed is not appropriate ? More generally, does it happen in the literature of statistics that the realization of $L$ happens to be greater than the realization of $U$ for some really critical sample outcomes ?


$$-----------------------------------------------------$$ Edit:

enter image description here

I added a picture of definition of confidence interval and a clarifying paragraph about it. Question: Let a sample be drawn and the realization of the random variables $L,U$ of the $1-\alpha$ confidence interval turns out to be $l,u$. What happens if $l$ happened to be strictly greater than greater than $u$ ? How does the statistician interpret the result ?

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Edit: One of the answers asked me to clarify my use of the inclusion exclusion principle:

$$P(\overline{X}-\frac{1}{2\sqrt{k\alpha}}<p<\overline{X}+\frac{1}{2\sqrt{k\alpha}})\geq 1-\frac{\alpha}{2}$$ $$P(\overline{Y}-\frac{1}{\sqrt{2k\alpha}}<p<\overline{Y}+\frac{1}{\sqrt{2k\alpha}})\geq 1-\frac{\alpha}{2}$$ Denote the event $\{\overline{X}-\frac{1}{2\sqrt{k\alpha}}<p<\overline{X}+\frac{1}{2\sqrt{k\alpha}}\}$ by $A$, and denote the event $\{\overline{Y}-\frac{1}{\sqrt{2k\alpha}}<p<\overline{Y}+\frac{1}{\sqrt{2k\alpha}}\}$ by $B$ . Denote the event $\overline{Y}-\frac{1}{\sqrt{2k\alpha}}<p<\overline{X}+\frac{1}{2\sqrt{k\alpha}}$ by $C$.
The result follows from noting that $A\cap B\subseteq C$, hence:

$$P(C)\geq P(A\cap B)=P(A)+P(B)-P(A\cup B)\geq 1-\frac{\alpha}{2}+1-\frac{\alpha}{2}-1=1-\alpha$$

$\endgroup$
7
  • 3
    $\begingroup$ Your project is hopeless. Because the variables are iid, it is a mathematical theorem that the count of heads contains all the information. Research "sufficient statistic." $\endgroup$
    – whuber
    May 3, 2023 at 21:37
  • $\begingroup$ @whuber Thank you for your response, what about the answer to the other quesitons. Question 2 and 3 in particular $\endgroup$
    – Amr
    May 3, 2023 at 21:39
  • 3
    $\begingroup$ Please revise your post to ask just one focused question. But when you invent your own statistical procedure and discover paradoxical results, consider that you have done something erroneous. $\endgroup$
    – whuber
    May 3, 2023 at 21:44
  • 1
    $\begingroup$ @whuber I did consider that possiblity but cant see the error in my reasoning and would be happy if someone here points out to me. Thats why I asked my question $\endgroup$
    – Amr
    May 3, 2023 at 21:48
  • 1
    $\begingroup$ @SextusEmpiricus Nope, I am only assuming that $P(A\cup B) \leq 1$ $\endgroup$
    – Amr
    May 5, 2023 at 13:56

3 Answers 3

3
$\begingroup$

There is no contradiction here. If $l > u$ then the confidence interval for this experiment is empty. This may seem disturbing, but there is nothing in the definition to prevent it.

You can find more examples in the paper "Bayesian Intervals and Confidence Intervals" by Jaynes. Or even do this: after your experiment, take the empty interval with probability 0.01 and [0,1] with probability 0.99.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. Yes, I understand the defintion allows it. I thought that when the confidence interval is empty then the model should be thrown away, in spirit of falsfication of the scientific method. I can elaborate more on that if it is not clear. $\endgroup$
    – Amr
    May 4, 2023 at 23:51
  • $\begingroup$ I will check the paper you refrence $\endgroup$
    – Amr
    May 4, 2023 at 23:52
2
$\begingroup$

Your confidence interval (CI) in the example obviously suffers from the construction issue that observations $X_2, X_4, X_6,\ldots$ are given much more weight through $\bar Y$ than $X_1, X_3,\ldots$. Now $\bar Y=1$ will mean that the confidence interval concentrates on the upper half of probabilities, as the fact that the mean of $X_1, X_3,\ldots$ is zero is (mostly) ignored. Even though you defined a technically valid confidence interval, it isn't a good one as optimal use of the information in the data can be made by having the CI dependent on the sufficient statistic $\bar X$ only; the involvement of $\bar Y$ just adds some "noise".

Looking at the data, you get an empty confidence interval. Obviously nothing in the definition of CIs stops this from happening, even if the model assumptions are fulfilled, so it doesn't necessarily indicate that model assumptions are violated. The answer of @Flounderer presents a CI that can be empty without giving any information about the data, including whether model assumptions are fulfilled, so in general an empty CI will not indicate that assumptions are not fulfilled. However...

Should the interpretation be that our model that $X_1, X_2,\ldots$ independent identically distributed is not appropriate?

All relevant characteristics of a CI are derived under the model assumptions, including the possibility of returning an empty interval. The standard theory of CIs doesn't indicate what happens if model assumptions are not fulfilled, therefore there is no reason in general to infer that model assumptions are violated in case an empty interval is returned.

However, looking at the specific definition, one can say something using the correspondence between CIs and tests. For a given CI (including the one in question), one can construct a test by rejecting any $H_0$ (i.e., here, Bernoulli probability $p_0$) that is not covered by the CI. We can also define a (quite conservative) test rejecting any $p_0$ just in case the CI is empty. In fact this test is a test of the i.i.d. Bernoulli(p)-model with arbitrary $p$, because it will reject with probability smaller than $\alpha$ (in fact I suspect the effective level is even $\le\frac{\alpha}{2}$ but I don't take the time to check or prove this) whenever the model holds with whatever $p$.

Now this also holds for the analogous test constructed from @Flounderer's trivial CI, but this CI in fact doesn't give any information about any violation of assumptions, as it has the same characteristics regardless of the underlying model (including if assumptions are violated in any way).

Your CI however is different. In fact it can be shown that there is a class of models under which the rejection probability, i.e., the power, is larger than $\alpha$. The test will reject with large probability if it is likely that $\bar Y$ is clearly larger than $\bar X$. This happens for example (and most prominently) if the data are not identically distributed, but (in order to make things easy) independently, so that there is a probability $p_1$ for success in $X_1,X_2,X_3,\ldots$ and a probability $p_2$ for success in $X_2,X_4,X_6,\ldots$, and $p_2>p_1$ with a large enough difference, which will depend on $\alpha$ and $k$ (I won't figure this out precisely but it shouldn't be difficult to do it; chances are it's $p_2>p_1+\epsilon$ with $\epsilon\searrow 0$ for $k\to\infty$).

So you are right, in principle; the event that the CI is empty can be interpreted as an unbiased test of the i.i.d. model against a model in which $\bar Y$ can be expected to be systematically larger than $\bar X$, and the easiest way to define such a model is above.

Note that this relies on precise analysis of the characteristics of the specific CI under both the nominal model and a model for which this test is likely to reject, which under rejection can then be interpreted as a better fitting model as evidenced by the data. The CI of @Flounderer shows that in general this is not always possible.

Obviously in the given case, in the first place one should suspect that model assumptions are violated from looking at the data, not from the result of your CI, but anyway, one can legitimately say that the test defined by "CI empty" tests the i.i.d. Bernoulli-model against the non-i.i.d. model defined above, and therefore an empty CI could reject the i.i.d. Bernoulli in favour of the non-i.i.d. model above (not a general non-i.i.d. model though, as if for example $p_1>p_2$ you will see an empty interval even less often than under i.i.d.).

$\endgroup$
9
  • $\begingroup$ Sure, I am aware I ignored the odd instances of the experiment when obtaining a lower bound. However, as you admit this still counts as a valid choice of CI by definition of CI(s). How is it that a valid defintion of CI leads to a result like this and how should the result be interpreted $\endgroup$
    – Amr
    May 4, 2023 at 1:46
  • $\begingroup$ In other words, the same reasonning that underlies the standard CI (the one where all data is averaged with equal weights ) also underlies the CI which I designed (because my CI is valid as you admit) $\endgroup$
    – Amr
    May 4, 2023 at 1:49
  • $\begingroup$ @Amr I added two paragraphs that should address this. $\endgroup$ May 4, 2023 at 9:32
  • $\begingroup$ Thank you for your edit. I am still not convinced. Perhaps I am missing something too obvious. $\endgroup$
    – Amr
    May 4, 2023 at 21:21
  • $\begingroup$ Let me ask one last question to avoid wasting your time any further. Please check the edit added to my question below the horizontal line $\endgroup$
    – Amr
    May 4, 2023 at 22:02
0
$\begingroup$

Your main line of thought seems to be about the bounds U and L. Say that the bounds have the properties that $P[p > U] \geq \alpha/2$ and $P[p < L] \geq \alpha/2$ (this is not the same as your statements with the Chebyshev inequality). So what if we combine two independently generated bounds (or slightly related bounds)? Can we use the confidence interval $P[L\geq p \geq U] \leq 1-\alpha$ when L and U are based on different experiments/data?

You can do that, but the consequence will be that in some rare occasions when $U<L$, you get an empty confidence interval (which obviously will not be true, because $p$ needs to have at least some value).

The reason that this can occur is because the probability that the confidence interval contains the parameter relates to the experiment as the random unit. Some experiments will result in a confidence interval that contains the parameter other experiments will not. See also Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?

$\endgroup$
3
  • $\begingroup$ It would help if you could use notation consistently with the question, i.e., bars rather than hats, and the interval given there has the $\bar Y$ in the lower bound and the $\bar X$ in the upper bound. Also I think $P[p>U]\le \alpha/2$ is at stake, not "$>$" etc. Bounds in the question are not "independently generated" although this may not make a difference to the argument. For the moment I believe the argument given in the question is correct; if you want to convince me otherwise, I think there is some repair work to do first. $\endgroup$ May 5, 2023 at 12:25
  • $\begingroup$ @ChristianHennig I believe that the argument in the text is not well motivated in how the step from the Chebyshev inequalities is made to the confidence interval. If I have two ranges $$P[a < x<b] = 1- \alpha/2 \\ P[c<x<d] = 1- \alpha/2$$then I can not just combine this into $$P[a<x<d] = 1- \alpha$$ However, if I have $$P[x< a] = \alpha/2\\P[x>d] = \alpha/2$$ then I can use $$P[a \leq x \leq d] = 1- \alpha$$ $\endgroup$ May 5, 2023 at 12:41
  • 1
    $\begingroup$ I added an edit at the bottom of my post to explain the use of the inclsuion exclusion principle $\endgroup$
    – Amr
    May 5, 2023 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.