# Cramér-Rao bound when the samples come from two distributions

Is there a version of the Cramér-Rao bound when samples are independent but not identically distributed? More specifically, I am considering a sample set that is divided in two subsets, each subset containing i.i.d. samples from a different Bernoulli distribution, with samples of different subsets being also independent. It is known from which distribution each sample comes from.

Perhaps a better way to view this is: two sample sets are used (instead of one), each from a different distribution; and both give information about the parameter of interest.

A specific example would be estimating the difference (or some other function) of two probabilities (Bernoulli parameters). The observations are two sets $$S_i$$, $$i=1,2$$ of i.i.d. Bernoulli variables with parameters $$p_i$$ and sizes $$n_i$$. Denoting $$S_i = \{x_{i,1}, \ldots, x_{i,n_i}\}$$, $$i=1,2$$, a natural estimator of $$d = p_1-p_2$$ would be the difference of observed proportions: $$\hat d = \hat p_1 - \hat p_2,$$ $$\hat p_i = \frac{1}{n_i} \sum_{k=1}^{n_i}x_{i,k} \quad i=1,2.$$ How can we assess how good (how efficient) this estimator is? Is there a Cramér-Rao bound for this type of situation? Or can the efficiency of $$\hat d$$ be somehow derived from the efficiencies of the individual estimators $$\hat p_i$$?

• Isn't the variance of $\hat{d}$ just the sum of the variance of $\hat{p}_1$ and $\hat{p}_2$? And as a consequence, the limit of the variance of $\hat{d}$ is the sum of the limits of the variance of $\hat{p}_1$ and $\hat{p}_2$. Commented Feb 28 at 21:33
• @Sextus Yes, that is the variance of $\hat d$. But then what do we compare that variance with, in order to compute efficiency? Commented Feb 28 at 21:49
• You compare it to the bounds based on the Cramer Rao bounds of the $\hat{p}_i$? That's the lowest variance you can get for an unbiased estimate. Commented Feb 28 at 21:55

A general result will be difficult. Sometimes, eg if $$p_1$$ and $$p_2$$ were the same in your example and you wanted to estimate that common $$p$$, the bound will be lower than the separate bounds. In that setting I believe the bound on the precision of the common $$\hat p$$ will be the sum of the bounds on precisions of the individual $$\hat p_i$$. That is $$\mathrm{var}^{-1}[\hat p]\leq \mathrm{var}^{-1}[\hat p_1]+\mathrm{var}^{-1}[\hat p_2]\leq n_1I_1+n_2I_2$$ where $$I_i$$ is the per-observation Fisher information for $$p$$ in sample $$i$$.

In your example, though, you can't get a bound for $$d$$ in the combined data from bounds for $$d$$ in the two separate samples because $$d$$ isn't identifiable from a single sample. You'll need something that considers them together. (well, for $$p_1-p_2$$ you could work it out, but general functions would be hard)

An asymptotic version would be more tractable. Write the data as $$(i, S_{i,n})$$ for $$i=1,2$$, $$n=1,\dots,n_i$$. It doesn't matter asymptotically whether $$n_i$$ are fixed or whether each observation is independently assigned to a set, so we can treat $$(I, S_{I,n})$$ as iid again, where $$I$$ is Bernoulli($$q$$) for some $$q$$. Now, $$d$$ is some differentiable function of $$(q,p_1,p_2)$$ and we can write down the Fisher information about $$d$$ and invoke the usual efficiency results -- one of the Convolution theorems or the local asymptotic minimax theorem. It doesn't matter whether $$\hat p_i$$ or $$\hat d$$ are finite-sample unbiased, which is another simplification for asymptotics, since for non-linear functions $$d$$ there will often be no unbiased estimators.

I don't think there's a shortcut to working out the Fisher information for $$d$$, which is the key step and which handles the distinction between $$(\hat p_1,\hat p_2)\mapsto \hat p$$ and $$(\hat p_1,\hat p_2)\mapsto (\hat p_1-\hat p_2)$$.

• I think I found an answer, but I'm not completely sure it is correct. Can you please take a look and give me your opinion? Commented Mar 3 at 0:41

I think I found an answer for the specific case described in my question.

Consider $$m$$ independent, identically distributed samples from a Bernoulli distribution with parameter $$p_1$$, and a similarly defined collection of $$n$$ samples with parameter $$p_2$$. Samples are also independent across the two different groups. Let $$d = g(p_1, p_2)$$.

The information matrix for $$[p_1 \ p_2]$$ is in this case $$\begin{equation*} \mathrm{\mathbf I} = \begin{bmatrix} \frac m {p_1(1-p_1)} & 0 \\ 0 & \frac n {p_2(1-p_2)} \end{bmatrix}, \end{equation*}$$ and therefore $$\begin{equation*} \mathrm{\mathbf I}^{-1} = \begin{bmatrix} \frac {p_1(1-p_1)} m & 0 \\ 0 & \frac {p_2(1-p_2)} n \end{bmatrix}. \end{equation*}$$ Assuming that $$g$$ has partial derivatives, let $$\mathrm{\mathbf J}$$ denote its Jacobian matrix (row vector), $$\begin{equation*} \mathrm{\mathbf J} = \begin{bmatrix} {\partial g}/{\partial p_1} & {\partial g}/{\partial p_2}. \end{bmatrix} \end{equation*}$$

Then, the vector form of the Cramér-Rao bound [see for example Patrick Breheny's notes, 10-23, slide 10; or Steven M. Kay's book Fundamentals of Statistical Signal Processing: Estimation Theory, eq. (3.30)] can be applied to give the following inequality for any unbiased estimator of $$d$$: $$\mathrm{Var}\, [\hat d] \geq \mathrm{\mathbf J} \mathrm{\mathbf I}^{-1} \mathrm{\mathbf J}^\top.$$ With $$\mathrm{\mathbf I}^{-1}$$ and $$\mathrm{\mathbf J}$$ as above, $$\mathrm{Var}\, [\hat d] \geq \left(\frac{\partial g}{\partial p_1}\right)^2 \frac {p_1(1-p_1)} m + \left(\frac{\partial g}{\partial p_2}\right)^2 \frac {p_1(1-p_1)} n,$$ which is the sought bound.

In the particular case that $$g(p_1,p_2) = p_1-p_2$$, $$\mathrm{Var}\, [d] \geq \frac {p_1(1-p_1)} m + \frac {p_1(1-p_1)} n,$$ and thus the estimator of $$d=p_1-p_2$$ proposed in the question is efficient.