Which variance should I use

Question

Let $X_1,\ldots X_n \sim Bern(p)$, be random variables with Bernoulli distribution and $x_1,\ldots x_n$ the observed data. This distribution has $\sigma^2$ the variance. Suppose I choose for the parameter $p$ the estimate $\hat{p}=\frac{x_1+\ldots x_n}{n}$. I want to know how the estimator $\overline X_n=\frac{X_1+\ldots X_n}{n}$ fluctuates in order to analyze whether $\hat p$ is a good estimate or not. Then I will find the variance of this estimator (the less variance the better).

Question 1.

All my reasoning is correct so far?

Question 2.

I've seen a bunch of formulas and I don't know why we have so many different formulas for the same concept (variance) and which one I need to choose.

$var(\bar X)=\frac{\sigma^2}{n}=\frac{p(1-p)}{n}$

$var(\bar X) = \frac{s^2}{n}$, where $s=\frac{\sum_{i=1}^n (x_i-\bar X)^2}{n}$

$var(\hat p)=MSE(\hat p)-bias(\hat p,p)= \mathbb E[(p-\hat p)^2]-bias(\hat p,p)$

Answer 1

The $X_{1},...,X_{n}$ variables coming from the $Bernoulli(p)$ and the $x_{1},...,x_{n}$ are realizations of those variables.

The main question is to check how well your $\hat{p}$ estimates the true parameter $p$.

Most frequently we want to check what happens to such assumptions asymptotically as $n\rightarrow \infty$.

So, you define an estimator of the parameter $p$, as

$\bar{X}=\frac{X_{1}+X_{2}+...+X_{n}}{n}$

where it becomes an estimation of $p$, i.e $\hat{p}$, when you plug in the realizations $x_{i}$.

The $var(\bar{X})=\frac{\sigma^{2}}{n}$ is the variance estimation of the $\bar{X}$ when you have knowledge of the true variance of the distribution. You can check that this variance goes to $0$ as you increase your sample.

In case where you do not know the true variance, you have to calculate it through your sample so you have as you wrote $var(\bar{X})=\frac{s^{2}}{n}$

In the last case, might want to check how much, your estimation $\hat{p}$ differs from the true parameter $p$. You can check that by calculating the Mean Square Error $MSE$, which as you noted is equal to the variance plus the bias.

However, $X$ is an unbiased estimator, so the bias term is zero. And from previously you check that variance can be reduced as you increase the number of samples. So, the $MSE$ can be reduced by increasing your sample i.e you get better approximation as you increase your sample.

All, these formulas are connected with the $MSE$ which you can use to check how close you approximate the true parameter $p$, so I guess you have to use everything that it is contained in the $MSE$ formula.