# Obtaining confidence intervals from composite defined functions

Assume you have a set of $$n$$ independent random variables $$X_1, X_2, \dots, X_n$$ with unknown distribution and mean (finite) values $$\mu_1, \mu_2, \dots, \mu_n \in \mathbb{R}$$. Moreover, there are $$n$$ known probabilities $$p_1, p_2, \dots, p_n$$ with $$p_i > 0$$ and $$p_1 + p_2 + \dots + p_n = 1$$. Upon it, we construct a random variable $$X$$ defined as: $$X = \begin{cases} X_1 & \text{with probability }p_1\\ X_2 & \text{with probability }p_2\\ \dots\\ X_n & \text{with probability }p_n \end{cases}$$ So, basically speaking, $$X$$ (which also has an unknown finite mean $$\mu$$) takes the value from one random variable $$X_1, X_2, ..., X_n$$ with a certain probability.

We can create i.i.d. $$k$$ samples $$x_1, x_2, \dots, x_k$$ from $$X$$ and can calculate a confidence interval for the mean $$\mu$$, e.g., by using the Student's $$t$$ distribution.

The question is now the following: Assume that we do not sample directly from $$X$$, but take $$k_1$$ samples from $$X_1$$, $$k_2$$ samples from $$X_2$$, and so on. We can calculate for each individual random variable $$X_1, X_2, \dots, X_n$$ the confidence intervals for the means $$\mu_1, \mu_2, \mu_3$$, but can we also compute for $$X$$ a confidence interval for the mean $$\mu$$ from these $$k_1 + k_2 + \dots + k_n$$ samples, as these are not i.i.d. samples for $$X$$ anymore?

Example: We know beforehand that $$n = 3$$ and $$p_1 = 0.2, p_2 = 0.4, p_3 = 0.4$$. Furthermore, we assume that each $$X_i$$ is Bernoulli distributed with an unknown mean $$\mu_i$$, so it is either $$0$$ or $$1$$ with an unknown probability. When we sample directly from $$X$$, each sample is i.i.d., and we do not know if $$X_1$$, $$X_2$$, or $$X_3$$ is chosen as the underlying random variable, making it easy to construct a confidence interval for the mean $$\mu$$. When we sample directly each $$X_i$$ $$2$$ times, we could, e.g., observe: $$X_1$$: $$x_{1,1} = 1, x_{1,2} = 0$$, $$X_2$$: $$x_{2,1} = 0, x_{2,2} = 1$$, $$X_3$$: $$x_{3,1} = 0, x_{3,2} = 0$$. As the number of samples for each $$X_i$$ is now fixed, the samples are no longer i.i.d.

• There are $n-1$ functionally independent parameters in your model. For what parameter are you constructing the confidence interval? (Your assumptions do not even imply that $X$ has a mean.) BTW, the distribution of $X$ is a (finite) mixture with weights $p_i.$
– whuber
Commented Jan 13 at 16:34
• Thanks for your reply! My assumption is that $p_1, p_2, ... p_n$ are known beforehand (maybe I didn't made that clear), but for $X_1, X_2, ..., X_n$ only a sampler exists. Commented Jan 14 at 10:50
• Example: We know beforehand that $n = 3$ and $p_1 = 0.2, p_2 = 0.4, p_3 = 0.4$. Furthermore, we assume that each $X_i$ is Bernoulli distributed, so it is either $0$ or $1$ with an unknown probability. After that, we sample each $X_i$ $2$ times, thus we observe $X_1$: $x_{1,1} = 1, x_{1,2} = 0$, $X_2$: $x_{2,1} = 0, x_{2,2} = 1$, $X_3$: $x_{3,1} = 0, x_{3,2} = 0$. Can we now construct a confidence interval for $X$? Commented Jan 14 at 10:58
• "Confidence interval for $X$" makes no sense. Please review the definitions.
– whuber
Commented Jan 14 at 16:23
• Thanks for pointing that out. I've updated my post accordingly: The rephrased question is whether we can build a confidence interval for the mean of $X$, $\mu\in\mathbb{R}$, as indicated in the post. Commented Jan 14 at 20:51

I guess that I have found one solution when we are using the central limit theorem to derive confidence intervals: We calculate a confidence interval for the mean $$\mu$$ when using the estimator $$\overline{X} = \sum_{i=1}^{n} \frac{p_i}{k_i} \cdot \sum_{j=1}^{k_i} x_{i,j}\text{.}$$ When assuming that the individual number of samples $$k_i$$ are chosen large enough, we know that the expression $$\frac{1}{k_i} \cdot \sum_{j=1}^{k_i} x_{i,j}$$ converges to a normal distribution $$\mathcal{N}\left(\mu_i, \frac{\sigma^2_i}{k_i}\right)$$. We assume that both the mean $$\mu_i$$ and the variance $$\sigma^2_i$$ of $$X_i$$ exist and are finite. Furthermore, we either know $$\mu_i$$ beforehand or have to estimate it. In the latter case, the Student's $$t$$-distribution might be more suitable, but for a large enough sample size the difference to the standard normal distribution is neglectable.
Next, we recall that for two independent random variables $$Y\sim \mathcal{N}(\mu_{Y}, \sigma_{Y}^2)$$ and $$Z\sim \mathcal{N}(\mu_{Z}, \sigma_{Z}^2)$$ the sum $$Y + Z$$ is distributed accordingly to $$\mathcal{N}(\mu_{Y} + \mu_{Z}, \sigma_{Y}^2 + \sigma_{Z}^2)$$ (see https://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables), and $$\text{Var}(pX) = p^2\text{Var}(X)$$. Using this information, we know that $$\overline{X}$$ converges to $$\mathcal{N}\left(\sum_{i=1}^{n} p_i\cdot \mu_i, \sum_{i=1}^{n} \frac{p_i^2}{k_i}\cdot \sigma^2_i\right)\text{.}$$ Now, we can do the same reformulation as done for the regular derivation of the confidence interval for the mean (see https://en.wikipedia.org/wiki/Confidence_interval#Example), and obtain $$\left[\overline{X} - z_{\alpha/2}\sqrt{\sum_{i=1}^{n} \frac{p_i^2}{k_i}\cdot \sigma^2_i}, \overline{X} + z_{\alpha/2}\sqrt{\sum_{i=1}^{n} \frac{p_i^2}{k_i}\cdot \sigma^2_i}\right]$$ as $$1 - \alpha$$ confidence interval for $$\overline{X}$$, with $$z_{\alpha/2}$$ being the $$\alpha/2$$-quantile for the standard normal distribution.
As open question remains, if we can proceed similarly when we have calculated the confidence interval for the mean with, e.g., the Student's $$t$$-distribution for an unknown variance or the Wilson score interval for Bernoulli distributions.