Paradoxical Questions about confidence intervals

Question

Let there be a repeatable real world experiment with two outcomes denoted by $0,1$ for convenience (Tossing a coin for example). Let $X_i$ be the random variable that models the ith repetition of the experiment. It is an assumption of our model of that real world phenomenon that $X_1,X_2,X_3,...$ are independent identically distributed to $B(1,p)$. I noticed in all confidence intervals for $p$ I encountered so far, one basically throws away all information in the sample and only keeps track of the total number of 1(s) (Total number of heads in case of a coin).

I came up with the following confidence interval. First let me make clear my defintion of a confidence interval in case we are working with a sample of size $n$.

Defintion: A $1-\alpha$ confidence interval for the parameter $p$ above are random variables $L,U$ that are functions of our sample $X_1,X_2,...,X_n$ such that $P(L<p<U)\geq 1-\alpha$. It is almost similar to the defintion of my book.

Using this defintion, I proceed to design my own confidence interval. Suppose our sample size is even of size $n=2k$. Set $\overline{X}$ to be the average of $X_1,X_2,X_3,...,X_{2k}$, and set $\overline{Y}$ to be the average of $X_2,X_4,X_6,...,X_{2k}$ By Chebyshev inequality, we have the inequalities below: $$P(\overline{X}-\frac{1}{2\sqrt{k\alpha}}<p<\overline{X}+\frac{1}{2\sqrt{k\alpha}})\geq 1-\frac{\alpha}{2}$$ $$P(\overline{Y}-\frac{1}{\sqrt{2k\alpha}}<p<\overline{Y}+\frac{1}{\sqrt{2k\alpha}})\geq 1-\frac{\alpha}{2}$$ By Inclusion exclusion principle, we get: $$P(\overline{Y}-\frac{1}{\sqrt{2k\alpha}}<p<\overline{X}+\frac{1}{2\sqrt{k\alpha}})\geq 1-\alpha$$. Thus, we get a $(1-\alpha)$ confidence interval which is $]\overline{Y}-\frac{1}{\sqrt{2k\alpha}},\overline{X}+\frac{1}{2\sqrt{k\alpha}}[$ Ofourse one could even consider more interesting statistics (more interesting than $\overline{Y}$)from the sample like for example the number of occurrences of the strings $1,0,0,0,1$ in the data collected.

Question : Now suppose we use the above confidence interval with confidence 99% for the case of tossing a coin $2\times 10^{30}$ times and it happens that we get the sample realization $0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,1,0,.....$, then applying the confidence interval above that I designed will give approximately something like $0.999...<p<0.50001$. I am not sure how to interpret the result of this confidence interval in this case. Should the interpretation be that our model that $X_1,X_2,...$ independent identically distributed is not appropriate ? More generally, does it happen in the literature of statistics that the realization of $L$ happens to be greater than the realization of $U$ for some really critical sample outcomes ?

---

$$-----------------------------------------------------$$ Edit:

I added a picture of definition of confidence interval and a clarifying paragraph about it. Question: Let a sample be drawn and the realization of the random variables $L,U$ of the $1-\alpha$ confidence interval turns out to be $l,u$. What happens if $l$ happened to be strictly greater than greater than $u$ ? How does the statistician interpret the result ?

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Edit: One of the answers asked me to clarify my use of the inclusion exclusion principle:

$$P(\overline{X}-\frac{1}{2\sqrt{k\alpha}}<p<\overline{X}+\frac{1}{2\sqrt{k\alpha}})\geq 1-\frac{\alpha}{2}$$ $$P(\overline{Y}-\frac{1}{\sqrt{2k\alpha}}<p<\overline{Y}+\frac{1}{\sqrt{2k\alpha}})\geq 1-\frac{\alpha}{2}$$ Denote the event $\{\overline{X}-\frac{1}{2\sqrt{k\alpha}}<p<\overline{X}+\frac{1}{2\sqrt{k\alpha}}\}$ by $A$, and denote the event $\{\overline{Y}-\frac{1}{\sqrt{2k\alpha}}<p<\overline{Y}+\frac{1}{\sqrt{2k\alpha}}\}$ by $B$ . Denote the event $\overline{Y}-\frac{1}{\sqrt{2k\alpha}}<p<\overline{X}+\frac{1}{2\sqrt{k\alpha}}$ by $C$.
The result follows from noting that $A\cap B\subseteq C$, hence:

$$P(C)\geq P(A\cap B)=P(A)+P(B)-P(A\cup B)\geq 1-\frac{\alpha}{2}+1-\frac{\alpha}{2}-1=1-\alpha$$

Answer 1

There is no contradiction here. If $l > u$ then the confidence interval for this experiment is empty. This may seem disturbing, but there is nothing in the definition to prevent it.

You can find more examples in the paper "Bayesian Intervals and Confidence Intervals" by Jaynes. Or even do this: after your experiment, take the empty interval with probability 0.01 and [0,1] with probability 0.99.

Answer 2

Your confidence interval (CI) in the example obviously suffers from the construction issue that observations $X_2, X_4, X_6,\ldots$ are given much more weight through $\bar Y$ than $X_1, X_3,\ldots$. Now $\bar Y=1$ will mean that the confidence interval concentrates on the upper half of probabilities, as the fact that the mean of $X_1, X_3,\ldots$ is zero is (mostly) ignored. Even though you defined a technically valid confidence interval, it isn't a good one as optimal use of the information in the data can be made by having the CI dependent on the sufficient statistic $\bar X$ only; the involvement of $\bar Y$ just adds some "noise".

Looking at the data, you get an empty confidence interval. Obviously nothing in the definition of CIs stops this from happening, even if the model assumptions are fulfilled, so it doesn't necessarily indicate that model assumptions are violated. The answer of @Flounderer presents a CI that can be empty without giving any information about the data, including whether model assumptions are fulfilled, so in general an empty CI will not indicate that assumptions are not fulfilled. However...

Should the interpretation be that our model that $X_1, X_2,\ldots$ independent identically distributed is not appropriate?

All relevant characteristics of a CI are derived under the model assumptions, including the possibility of returning an empty interval. The standard theory of CIs doesn't indicate what happens if model assumptions are not fulfilled, therefore there is no reason in general to infer that model assumptions are violated in case an empty interval is returned.

However, looking at the specific definition, one can say something using the correspondence between CIs and tests. For a given CI (including the one in question), one can construct a test by rejecting any $H_0$ (i.e., here, Bernoulli probability $p_0$) that is not covered by the CI. We can also define a (quite conservative) test rejecting any $p_0$ just in case the CI is empty. In fact this test is a test of the i.i.d. Bernoulli(p)-model with arbitrary $p$, because it will reject with probability smaller than $\alpha$ (in fact I suspect the effective level is even $\le\frac{\alpha}{2}$ but I don't take the time to check or prove this) whenever the model holds with whatever $p$.

Now this also holds for the analogous test constructed from @Flounderer's trivial CI, but this CI in fact doesn't give any information about any violation of assumptions, as it has the same characteristics regardless of the underlying model (including if assumptions are violated in any way).

Your CI however is different. In fact it can be shown that there is a class of models under which the rejection probability, i.e., the power, is larger than $\alpha$. The test will reject with large probability if it is likely that $\bar Y$ is clearly larger than $\bar X$. This happens for example (and most prominently) if the data are not identically distributed, but (in order to make things easy) independently, so that there is a probability $p_1$ for success in $X_1,X_2,X_3,\ldots$ and a probability $p_2$ for success in $X_2,X_4,X_6,\ldots$, and $p_2>p_1$ with a large enough difference, which will depend on $\alpha$ and $k$ (I won't figure this out precisely but it shouldn't be difficult to do it; chances are it's $p_2>p_1+\epsilon$ with $\epsilon\searrow 0$ for $k\to\infty$).

So you are right, in principle; the event that the CI is empty can be interpreted as an unbiased test of the i.i.d. model against a model in which $\bar Y$ can be expected to be systematically larger than $\bar X$, and the easiest way to define such a model is above.

Note that this relies on precise analysis of the characteristics of the specific CI under both the nominal model and a model for which this test is likely to reject, which under rejection can then be interpreted as a better fitting model as evidenced by the data. The CI of @Flounderer shows that in general this is not always possible.

Obviously in the given case, in the first place one should suspect that model assumptions are violated from looking at the data, not from the result of your CI, but anyway, one can legitimately say that the test defined by "CI empty" tests the i.i.d. Bernoulli-model against the non-i.i.d. model defined above, and therefore an empty CI could reject the i.i.d. Bernoulli in favour of the non-i.i.d. model above (not a general non-i.i.d. model though, as if for example $p_1>p_2$ you will see an empty interval even less often than under i.i.d.).

Answer 3

Your main line of thought seems to be about the bounds U and L. Say that the bounds have the properties that $P[p > U] \geq \alpha/2$ and $P[p < L] \geq \alpha/2$ (this is not the same as your statements with the Chebyshev inequality). So what if we combine two independently generated bounds (or slightly related bounds)? Can we use the confidence interval $P[L\geq p \geq U] \leq 1-\alpha$ when L and U are based on different experiments/data?

You can do that, but the consequence will be that in some rare occasions when $U<L$, you get an empty confidence interval (which obviously will not be true, because $p$ needs to have at least some value).

The reason that this can occur is because the probability that the confidence interval contains the parameter relates to the experiment as the random unit. Some experiments will result in a confidence interval that contains the parameter other experiments will not. See also Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?