Cramér-Rao bound when the samples come from two distributions

Question

Is there a version of the Cramér-Rao bound when samples are independent but not identically distributed? More specifically, I am considering a sample set that is divided in two subsets, each subset containing i.i.d. samples from a different Bernoulli distribution, with samples of different subsets being also independent. It is known from which distribution each sample comes from.

Perhaps a better way to view this is: two sample sets are used (instead of one), each from a different distribution; and both give information about the parameter of interest.

A specific example would be estimating the difference (or some other function) of two probabilities (Bernoulli parameters). The observations are two sets $S_i$, $i=1,2$ of i.i.d. Bernoulli variables with parameters $p_i$ and sizes $n_i$. Denoting $S_i = \{x_{i,1}, \ldots, x_{i,n_i}\}$, $i=1,2$, a natural estimator of $d = p_1-p_2$ would be the difference of observed proportions: $$ \hat d = \hat p_1 - \hat p_2, $$ $$ \hat p_i = \frac{1}{n_i} \sum_{k=1}^{n_i}x_{i,k} \quad i=1,2. $$ How can we assess how good (how efficient) this estimator is? Is there a Cramér-Rao bound for this type of situation? Or can the efficiency of $\hat d$ be somehow derived from the efficiencies of the individual estimators $\hat p_i$?

Answer 1

A general result will be difficult. Sometimes, eg if $p_1$ and $p_2$ were the same in your example and you wanted to estimate that common $p$, the bound will be lower than the separate bounds. In that setting I believe the bound on the precision of the common $\hat p$ will be the sum of the bounds on precisions of the individual $\hat p_i$. That is $$\mathrm{var}^{-1}[\hat p]\leq \mathrm{var}^{-1}[\hat p_1]+\mathrm{var}^{-1}[\hat p_2]\leq n_1I_1+n_2I_2$$ where $I_i$ is the per-observation Fisher information for $p$ in sample $i$.

In your example, though, you can't get a bound for $d$ in the combined data from bounds for $d$ in the two separate samples because $d$ isn't identifiable from a single sample. You'll need something that considers them together. (well, for $p_1-p_2$ you could work it out, but general functions would be hard)

An asymptotic version would be more tractable. Write the data as $(i, S_{i,n})$ for $i=1,2$, $n=1,\dots,n_i$. It doesn't matter asymptotically whether $n_i$ are fixed or whether each observation is independently assigned to a set, so we can treat $(I, S_{I,n})$ as iid again, where $I$ is Bernoulli($q$) for some $q$. Now, $d$ is some differentiable function of $(q,p_1,p_2)$ and we can write down the Fisher information about $d$ and invoke the usual efficiency results -- one of the Convolution theorems or the local asymptotic minimax theorem. It doesn't matter whether $\hat p_i$ or $\hat d$ are finite-sample unbiased, which is another simplification for asymptotics, since for non-linear functions $d$ there will often be no unbiased estimators.

I don't think there's a shortcut to working out the Fisher information for $d$, which is the key step and which handles the distinction between $(\hat p_1,\hat p_2)\mapsto \hat p$ and $(\hat p_1,\hat p_2)\mapsto (\hat p_1-\hat p_2)$.

Answer 2

I think I found an answer for the specific case described in my question.

Consider $m$ independent, identically distributed samples from a Bernoulli distribution with parameter $p_1$, and a similarly defined collection of $n$ samples with parameter $p_2$. Samples are also independent across the two different groups. Let $d = g(p_1, p_2)$.

The information matrix for $[p_1 \ p_2]$ is in this case \begin{equation*} \mathrm{\mathbf I} = \begin{bmatrix} \frac m {p_1(1-p_1)} & 0 \\ 0 & \frac n {p_2(1-p_2)} \end{bmatrix}, \end{equation*} and therefore \begin{equation*} \mathrm{\mathbf I}^{-1} = \begin{bmatrix} \frac {p_1(1-p_1)} m & 0 \\ 0 & \frac {p_2(1-p_2)} n \end{bmatrix}. \end{equation*} Assuming that $g$ has partial derivatives, let $\mathrm{\mathbf J}$ denote its Jacobian matrix (row vector), \begin{equation*} \mathrm{\mathbf J} = \begin{bmatrix} {\partial g}/{\partial p_1} & {\partial g}/{\partial p_2}. \end{bmatrix} \end{equation*}

Then, the vector form of the Cramér-Rao bound [see for example Patrick Breheny's notes, 10-23, slide 10; or Steven M. Kay's book Fundamentals of Statistical Signal Processing: Estimation Theory, eq. (3.30)] can be applied to give the following inequality for any unbiased estimator of $d$: $$ \mathrm{Var}\, [\hat d] \geq \mathrm{\mathbf J} \mathrm{\mathbf I}^{-1} \mathrm{\mathbf J}^\top. $$ With $\mathrm{\mathbf I}^{-1}$ and $\mathrm{\mathbf J}$ as above, $$ \mathrm{Var}\, [\hat d] \geq \left(\frac{\partial g}{\partial p_1}\right)^2 \frac {p_1(1-p_1)} m + \left(\frac{\partial g}{\partial p_2}\right)^2 \frac {p_1(1-p_1)} n, $$ which is the sought bound.

In the particular case that $g(p_1,p_2) = p_1-p_2$, $$ \mathrm{Var}\, [d] \geq \frac {p_1(1-p_1)} m + \frac {p_1(1-p_1)} n, $$ and thus the estimator of $d=p_1-p_2$ proposed in the question is efficient.