1
$\begingroup$

Is there a version of the Cramér-Rao bound when samples are independent but not identically distributed? More specifically, I am considering a sample set that is divided in two subsets, each subset containing i.i.d. samples from a different Bernoulli distribution, with samples of different subsets being also independent. It is known from which distribution each sample comes from.

Perhaps a better way to view this is: two sample sets are used (instead of one), each from a different distribution; and both give information about the parameter of interest.

A specific example would be estimating the difference (or some other function) of two probabilities (Bernoulli parameters). The observations are two sets $S_i$, $i=1,2$ of i.i.d. Bernoulli variables with parameters $p_i$ and sizes $n_i$. Denoting $S_i = \{x_{i,1}, \ldots, x_{i,n_i}\}$, $i=1,2$, a natural estimator of $d = p_1-p_2$ would be the difference of observed proportions: $$ \hat d = \hat p_1 - \hat p_2, $$ $$ \hat p_i = \frac{1}{n_i} \sum_{k=1}^{n_i}x_{i,k} \quad i=1,2. $$ How can we assess how good (how efficient) this estimator is? Is there a Cramér-Rao bound for this type of situation? Or can the efficiency of $\hat d$ be somehow derived from the efficiencies of the individual estimators $\hat p_i$?

$\endgroup$
3
  • $\begingroup$ Isn't the variance of $\hat{d}$ just the sum of the variance of $\hat{p}_1$ and $\hat{p}_2$? And as a consequence, the limit of the variance of $\hat{d}$ is the sum of the limits of the variance of $\hat{p}_1$ and $\hat{p}_2$. $\endgroup$ Commented Feb 28 at 21:33
  • $\begingroup$ @Sextus Yes, that is the variance of $\hat d$. But then what do we compare that variance with, in order to compute efficiency? $\endgroup$
    – Luis Mendo
    Commented Feb 28 at 21:49
  • $\begingroup$ You compare it to the bounds based on the Cramer Rao bounds of the $\hat{p}_i$? That's the lowest variance you can get for an unbiased estimate. $\endgroup$ Commented Feb 28 at 21:55

2 Answers 2

2
$\begingroup$

A general result will be difficult. Sometimes, eg if $p_1$ and $p_2$ were the same in your example and you wanted to estimate that common $p$, the bound will be lower than the separate bounds. In that setting I believe the bound on the precision of the common $\hat p$ will be the sum of the bounds on precisions of the individual $\hat p_i$. That is $$\mathrm{var}^{-1}[\hat p]\leq \mathrm{var}^{-1}[\hat p_1]+\mathrm{var}^{-1}[\hat p_2]\leq n_1I_1+n_2I_2$$ where $I_i$ is the per-observation Fisher information for $p$ in sample $i$.

In your example, though, you can't get a bound for $d$ in the combined data from bounds for $d$ in the two separate samples because $d$ isn't identifiable from a single sample. You'll need something that considers them together. (well, for $p_1-p_2$ you could work it out, but general functions would be hard)

An asymptotic version would be more tractable. Write the data as $(i, S_{i,n})$ for $i=1,2$, $n=1,\dots,n_i$. It doesn't matter asymptotically whether $n_i$ are fixed or whether each observation is independently assigned to a set, so we can treat $(I, S_{I,n})$ as iid again, where $I$ is Bernoulli($q$) for some $q$. Now, $d$ is some differentiable function of $(q,p_1,p_2)$ and we can write down the Fisher information about $d$ and invoke the usual efficiency results -- one of the Convolution theorems or the local asymptotic minimax theorem. It doesn't matter whether $\hat p_i$ or $\hat d$ are finite-sample unbiased, which is another simplification for asymptotics, since for non-linear functions $d$ there will often be no unbiased estimators.

I don't think there's a shortcut to working out the Fisher information for $d$, which is the key step and which handles the distinction between $(\hat p_1,\hat p_2)\mapsto \hat p$ and $(\hat p_1,\hat p_2)\mapsto (\hat p_1-\hat p_2)$.

$\endgroup$
1
  • $\begingroup$ I think I found an answer, but I'm not completely sure it is correct. Can you please take a look and give me your opinion? $\endgroup$
    – Luis Mendo
    Commented Mar 3 at 0:41
0
$\begingroup$

I think I found an answer for the specific case described in my question.

Consider $m$ independent, identically distributed samples from a Bernoulli distribution with parameter $p_1$, and a similarly defined collection of $n$ samples with parameter $p_2$. Samples are also independent across the two different groups. Let $d = g(p_1, p_2)$.

The information matrix for $[p_1 \ p_2]$ is in this case \begin{equation*} \mathrm{\mathbf I} = \begin{bmatrix} \frac m {p_1(1-p_1)} & 0 \\ 0 & \frac n {p_2(1-p_2)} \end{bmatrix}, \end{equation*} and therefore \begin{equation*} \mathrm{\mathbf I}^{-1} = \begin{bmatrix} \frac {p_1(1-p_1)} m & 0 \\ 0 & \frac {p_2(1-p_2)} n \end{bmatrix}. \end{equation*} Assuming that $g$ has partial derivatives, let $\mathrm{\mathbf J}$ denote its Jacobian matrix (row vector), \begin{equation*} \mathrm{\mathbf J} = \begin{bmatrix} {\partial g}/{\partial p_1} & {\partial g}/{\partial p_2}. \end{bmatrix} \end{equation*}

Then, the vector form of the Cramér-Rao bound [see for example Patrick Breheny's notes, 10-23, slide 10; or Steven M. Kay's book Fundamentals of Statistical Signal Processing: Estimation Theory, eq. (3.30)] can be applied to give the following inequality for any unbiased estimator of $d$: $$ \mathrm{Var}\, [\hat d] \geq \mathrm{\mathbf J} \mathrm{\mathbf I}^{-1} \mathrm{\mathbf J}^\top. $$ With $\mathrm{\mathbf I}^{-1}$ and $\mathrm{\mathbf J}$ as above, $$ \mathrm{Var}\, [\hat d] \geq \left(\frac{\partial g}{\partial p_1}\right)^2 \frac {p_1(1-p_1)} m + \left(\frac{\partial g}{\partial p_2}\right)^2 \frac {p_1(1-p_1)} n, $$ which is the sought bound.

In the particular case that $g(p_1,p_2) = p_1-p_2$, $$ \mathrm{Var}\, [d] \geq \frac {p_1(1-p_1)} m + \frac {p_1(1-p_1)} n, $$ and thus the estimator of $d=p_1-p_2$ proposed in the question is efficient.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.