1
$\begingroup$

Let $X_1,\ldots X_n \sim Bern(p)$, be random variables with Bernoulli distribution and $x_1,\ldots x_n$ the observed data. This distribution has $\sigma^2$ the variance. Suppose I choose for the parameter $p$ the estimate $\hat{p}=\frac{x_1+\ldots x_n}{n}$. I want to know how the estimator $\overline X_n=\frac{X_1+\ldots X_n}{n}$ fluctuates in order to analyze whether $\hat p$ is a good estimate or not. Then I will find the variance of this estimator (the less variance the better).

Question 1.

All my reasoning is correct so far?

Question 2.

I've seen a bunch of formulas and I don't know why we have so many different formulas for the same concept (variance) and which one I need to choose.

$var(\bar X)=\frac{\sigma^2}{n}=\frac{p(1-p)}{n}$

$var(\bar X) = \frac{s^2}{n}$, where $s=\frac{\sum_{i=1}^n (x_i-\bar X)^2}{n}$

$var(\hat p)=MSE(\hat p)-bias(\hat p,p)= \mathbb E[(p-\hat p)^2]-bias(\hat p,p)$

$\endgroup$
4
  • $\begingroup$ The first two formulas for variance differ in the meaning, that first is probabilistic, and second is stochastic version. In first we use $\sigma$ and $p$, which are known from theory (assumptions). In the second we aim to infer the variance from what we observe. Third formula is ment to show something in the special case of the variance, but it is hard to tell exactly what without some context. $\endgroup$
    – cure
    Dec 19 '20 at 12:57
  • $\begingroup$ If you look into, say, a physics text you will see many different formulas for forces. The point is that the situation matters: you shouldn't expect there to be a universal mathematical formula for every concept. By analogy, asking which force formula to choose is equivalent to asking how do you solve physics problems involving forces: it's too broad to be answerable on a physics site; and your question 2 is too broad to be answerable here. $\endgroup$
    – whuber
    Dec 19 '20 at 16:07
  • $\begingroup$ @whuber The answer to my question 1 is yes? in the question 2, the second formula is equivalent to second one in the case I mentioned? $\endgroup$
    – user45523
    Dec 19 '20 at 20:48
  • $\begingroup$ @cure The context is the one I mentioned in the question. $\endgroup$
    – user45523
    Dec 19 '20 at 20:51
1
$\begingroup$

The $X_{1},...,X_{n}$ variables coming from the $Bernoulli(p)$ and the $x_{1},...,x_{n}$ are realizations of those variables.

The main question is to check how well your $\hat{p}$ estimates the true parameter $p$.

Most frequently we want to check what happens to such assumptions asymptotically as $n\rightarrow \infty$.

So, you define an estimator of the parameter $p$, as

$\bar{X}=\frac{X_{1}+X_{2}+...+X_{n}}{n}$

where it becomes an estimation of $p$, i.e $\hat{p}$, when you plug in the realizations $x_{i}$.

The $var(\bar{X})=\frac{\sigma^{2}}{n}$ is the variance estimation of the $\bar{X}$ when you have knowledge of the true variance of the distribution. You can check that this variance goes to $0$ as you increase your sample.

In case where you do not know the true variance, you have to calculate it through your sample so you have as you wrote $var(\bar{X})=\frac{s^{2}}{n}$

In the last case, might want to check how much, your estimation $\hat{p}$ differs from the true parameter $p$. You can check that by calculating the Mean Square Error $MSE$, which as you noted is equal to the variance plus the bias.

However, $X$ is an unbiased estimator, so the bias term is zero. And from previously you check that variance can be reduced as you increase the number of samples. So, the $MSE$ can be reduced by increasing your sample i.e you get better approximation as you increase your sample.

All, these formulas are connected with the $MSE$ which you can use to check how close you approximate the true parameter $p$, so I guess you have to use everything that it is contained in the $MSE$ formula.

$\endgroup$
4
  • $\begingroup$ Thank you for your answer. The second formula is equivalent to the second one in the case I mentioned? $\endgroup$
    – user45523
    Dec 19 '20 at 20:50
  • $\begingroup$ Yes, it is equal to the samples variance divided by the number of samples $\endgroup$
    – Fiodor1234
    Dec 19 '20 at 22:55
  • $\begingroup$ Thank you again, last question I've seen some books doing this: they replace $\sigma^2=p(1-p)$ by $\hat p(1-\hat p)$, then $var(\bar X)=\hat p(1-\hat p)/n$. So should I use this formula or $var(\bar X)=s^2/n$? $\endgroup$
    – user45523
    Dec 20 '20 at 8:06
  • $\begingroup$ if you calculate the $\hat{p}(1-\hat{p})$ you will find that this is equal to $s^{2}$. So, when they do not know the true parameter $p$ or the true variance, they replace it with the estimated ones. This can be viewed in that way because the variance $\sigma^{2}$ can be expressed in terms of the true $p$ $\endgroup$
    – Fiodor1234
    Dec 20 '20 at 11:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.