I want to calculate the mean estimator and its variance (standard error) for the following example: Let's say I'm examining several patients per day and they are diagnosed with Cancer or No cancer. There were $n$ days in total:

Day 1: 100 patients examined, 4 were positive (cancer)

Day 2: 110 patients examined, 0 were positive


Day n: 105 patients examined, 3 were positive

I see two different ways of finding my estimator for mean (probability $p$) and standard error:

Case 1: I estimate a different probability for each day $p_x$ and then I average that:

$\hat{p} = \frac{p_1 + p_2 + ... + p_n}{n} = \frac{ \frac{4}{100} + \frac{0}{110} + ... + \frac{3}{105} }{n}$

Then the standard error (variance) is:

$var(\hat{p}) = var(\frac{1}{n}(var_1+var_2+...+var_n)) = \\ \frac{1}{n^2}(var_1+var_2+...+var_n) $

Here the $var_x$ for each day is $p_x(1-p_x)/n_x$, where $n_x$ denotes the number of patients examined per day. So my standard error is :

$var(\hat{p}) = \frac{1}{n^2}(\frac{\frac{4}{100}(1-\frac{4}{100})}{100}+...+\frac{\frac{3}{105}(1-\frac{3}{105})}{105}) $

Case 2: I merge all samples into one assuming that my probability $p$ is the same (series of Bernoulli experiments with same $p$). Then:

$\hat{p} = \frac{4+0+...+3}{100+110+...+105}$

$var(\hat{p}) = \frac{\hat{p}(1-\hat{p})}{100+110+...+105}$

The result is different using each case -- what's the right way of doing this?


There are two possible sources for the discrepancy:

  1. The number of patients varies from day to day, so you really should use a weighted mean. Then you should get the same answer with both metods! Try it.
  2. The true probability $p$ might vary amoung days, by analyzing data in the day by day method you get the possibility to investigate this. By merging all the samples you cannot investigate that.

Some details: Let the days be indexed by $i=1, \dots, n$ and the observation from day $i$ be $X_i \sim \text{Bin}(n_i,p_i)$, with days being independent (another possibility is a time series structure with time dependence, I will not discuss that). We have $X_i \sim \text{Bin}(n_i,p_i)$ and the estimator based on day $i$ is $\hat{p}_i=\frac{X_i}{n_i}$ which is an unbiased estimator for $p_i$ with variance $\frac{p_i (1-p_i)}{n_i}$. If we merge all the data together we get a merged estimator $\hat{p}=\frac{\sum X_i}{\sum n_i}$. If we assume that the probability is the same all the days, that is, that $p_i \equiv p$ where $p$ is the common value, the the merged estimator is unbiased as an estimator of the common $p$. Write $N=\sum n_i$.

Now let us look at the weighted average of all the $\hat{p}_i$. It is given by $$ \frac{\sum n_i \hat{p}_i}{\sum n_i}=\frac{\sum n_i \frac{X_i}{n_i}}{\sum n_i} = \frac{\sum X_i}{\sum n_i}= \frac{X}{N} $$ so you can see it is exactly the same as the estimator based on merged data.

So probably the difference you observed stems from you using an unweighted mean.

(We did'nt discuss whats happening if the $p_i$'s are not all equal)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.