Obtaining confidence intervals from composite defined functions

Question

Assume you have a set of $n$ independent random variables $X_1, X_2, \dots, X_n$ with unknown distribution and mean (finite) values $\mu_1, \mu_2, \dots, \mu_n \in \mathbb{R}$. Moreover, there are $n$ known probabilities $p_1, p_2, \dots, p_n$ with $p_i > 0$ and $p_1 + p_2 + \dots + p_n = 1$. Upon it, we construct a random variable $X$ defined as: $$X = \begin{cases} X_1 & \text{with probability }p_1\\ X_2 & \text{with probability }p_2\\ \dots\\ X_n & \text{with probability }p_n \end{cases}$$ So, basically speaking, $X$ (which also has an unknown finite mean $\mu$) takes the value from one random variable $X_1, X_2, ..., X_n$ with a certain probability.

We can create i.i.d. $k$ samples $x_1, x_2, \dots, x_k$ from $X$ and can calculate a confidence interval for the mean $\mu$, e.g., by using the Student's $t$ distribution.

The question is now the following: Assume that we do not sample directly from $X$, but take $k_1$ samples from $X_1$, $k_2$ samples from $X_2$, and so on. We can calculate for each individual random variable $X_1, X_2, \dots, X_n$ the confidence intervals for the means $\mu_1, \mu_2, \mu_3$, but can we also compute for $X$ a confidence interval for the mean $\mu$ from these $k_1 + k_2 + \dots + k_n$ samples, as these are not i.i.d. samples for $X$ anymore?

Example: We know beforehand that $n = 3$ and $p_1 = 0.2, p_2 = 0.4, p_3 = 0.4$. Furthermore, we assume that each $X_i$ is Bernoulli distributed with an unknown mean $\mu_i$, so it is either $0$ or $1$ with an unknown probability. When we sample directly from $X$, each sample is i.i.d., and we do not know if $X_1$, $X_2$, or $X_3$ is chosen as the underlying random variable, making it easy to construct a confidence interval for the mean $\mu$. When we sample directly each $X_i$ $2$ times, we could, e.g., observe: $X_1$: $x_{1,1} = 1, x_{1,2} = 0$, $X_2$: $x_{2,1} = 0, x_{2,2} = 1$, $X_3$: $x_{3,1} = 0, x_{3,2} = 0$. As the number of samples for each $X_i$ is now fixed, the samples are no longer i.i.d.

Answer 1

I guess that I have found one solution when we are using the central limit theorem to derive confidence intervals: We calculate a confidence interval for the mean $\mu$ when using the estimator $$\overline{X} = \sum_{i=1}^{n} \frac{p_i}{k_i} \cdot \sum_{j=1}^{k_i} x_{i,j}\text{.}$$ When assuming that the individual number of samples $k_i$ are chosen large enough, we know that the expression $\frac{1}{k_i} \cdot \sum_{j=1}^{k_i} x_{i,j}$ converges to a normal distribution $\mathcal{N}\left(\mu_i, \frac{\sigma^2_i}{k_i}\right)$. We assume that both the mean $\mu_i$ and the variance $\sigma^2_i$ of $X_i$ exist and are finite. Furthermore, we either know $\mu_i$ beforehand or have to estimate it. In the latter case, the Student's $t$-distribution might be more suitable, but for a large enough sample size the difference to the standard normal distribution is neglectable.

Next, we recall that for two independent random variables $Y\sim \mathcal{N}(\mu_{Y}, \sigma_{Y}^2)$ and $Z\sim \mathcal{N}(\mu_{Z}, \sigma_{Z}^2)$ the sum $Y + Z$ is distributed accordingly to $\mathcal{N}(\mu_{Y} + \mu_{Z}, \sigma_{Y}^2 + \sigma_{Z}^2)$ (see https://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables), and $\text{Var}(pX) = p^2\text{Var}(X)$. Using this information, we know that $\overline{X}$ converges to $$\mathcal{N}\left(\sum_{i=1}^{n} p_i\cdot \mu_i, \sum_{i=1}^{n} \frac{p_i^2}{k_i}\cdot \sigma^2_i\right)\text{.}$$ Now, we can do the same reformulation as done for the regular derivation of the confidence interval for the mean (see https://en.wikipedia.org/wiki/Confidence_interval#Example), and obtain $$\left[\overline{X} - z_{\alpha/2}\sqrt{\sum_{i=1}^{n} \frac{p_i^2}{k_i}\cdot \sigma^2_i}, \overline{X} + z_{\alpha/2}\sqrt{\sum_{i=1}^{n} \frac{p_i^2}{k_i}\cdot \sigma^2_i}\right]$$ as $1 - \alpha$ confidence interval for $\overline{X}$, with $z_{\alpha/2}$ being the $\alpha/2$-quantile for the standard normal distribution.

As open question remains, if we can proceed similarly when we have calculated the confidence interval for the mean with, e.g., the Student's $t$-distribution for an unknown variance or the Wilson score interval for Bernoulli distributions.