5
$\begingroup$

Given $X_1,X_2...X_n \sim F$ and $(\hat{F}_n(x))$ is an empirical CDF

I need to prove that $$Var(\hat{F}_n(x)) = \frac{F(x)(1 - F(x))}{n}$$

So what I did is:

The variance of an estimator $\hat{F}_n(x)$ of a population CDF $F(x)$ is this: (if I'm not mistaken)

$$Var(\hat{F}_n(x)) = E[(\hat{F}_n(x) - E[\hat{F}_n(x)])^2]$$

Since $\hat{F}_n(x)$ is an estimator of the population CDF $F(x)$, we can substitute $F(x)$ for $E[\hat{F}_n(x)]$ in the above equation to get:

$$Var(\hat{F}_n(x)) = E[(\hat{F}_n(x) - F(x))^2]$$

We can then use the fact that $\hat{F}_n(x)$ is a Bernoulli random variable with probability of success equal to $F(x)$ to expand the square and simplify the expression:

$$Var(\hat{F}_n(x)) = E[\hat{F}_n(x)^2 - 2\hat{F}_n(x)F(x) + F(x)^2]$$

Since it's a bernoulli we get that $\hat{F}_n(x)^2 = \hat{F}_n(x)$ and we know that $E[\hat{F}_n(x)] = F(x)$, therefore we plug the new values:

$$Var(\hat{F}_n(x)) = F(x) - 2F(x)^2 + F(x)^2 = F(x)(1 - F(x))$$

And this is where I get to a dead end, I get this formula but what I need to prove is this formula but with the right side divided by $n$, and I didn't get this. What did I do wrong?

EDIT-

After reading your comments I tried adjusting the approach a little bit, I did the following, I hope I'm right:

First we find the variance of $\mathop{\hat{F}_n}\left(x\right)$.

The variance of a random variable is defined as $Var(X) = E[(X - E[X])^2]$.

Substituting $\mathop{\hat{F}_n}\left(x\right)$ for $X$, we have:

$$Var(\hat{F}_n(x)) = E[(\hat{F}_n(x) - E[\hat{F}_n(x)])^2]$$

We can rewrite $\hat{F}_n(x)$ as $\frac{R_n(x)}{n}$, where $R_n(x) \sim \text{Binomial}(n, F(x))$, and that is because empirical CDF is the average of Bernoulli random variables. Therefore,

$$Var(\hat{F}_n(x)) = E\left[\left(\frac{R_n(x)}{n} - E\left[\frac{R_n(x)}{n}\right]\right)^2\right]$$

$$Var(\hat{F}_n(x)) = E\left[\frac{R_n(x)^2}{n^2}\right] - \left(E\left[\frac{R_n(x)}{n}\right]\right)^2$$

Since $R_n(x)$ is a binomial random variable, we can use the formula for the variance of a binomial random variable, which is $Var(R_n(x)) = nF(x)(1 - F(x))$. Therefore,

$$Var(\hat{F}_n(x)) = \frac{nF(x)(1 - F(x))}{n^2} - \left(\frac{nF(x)}{n}\right)^2$$

After simplifying we get:

$$Var(\hat{F}_n(x)) = \frac{F(x)(1 - F(x))}{n}$$

$\endgroup$
5
  • 3
    $\begingroup$ Recall the definition of the ecdf, which is an average of Bernoulli r.v.s $\endgroup$ Jan 3, 2023 at 10:43
  • $\begingroup$ Stating that "$\hat{F}_n(x)$ is an estimator of the population CDF $F(x)$" is not enough to conclude that "we can substitute $F(x)$ for $E[\hat{F}_n(x)]$". $\endgroup$
    – statmerkur
    Jan 3, 2023 at 11:00
  • $\begingroup$ @ChristophHanck So what should I add to my solution? I'm missing the point. $\endgroup$
    – CORy
    Jan 3, 2023 at 11:28
  • 2
    $\begingroup$ Basically what the answers that have come in by now say - recall that $Var(aX)=a^2Var(X)$. $\endgroup$ Jan 3, 2023 at 13:11
  • 1
    $\begingroup$ Because $n\hat F(x)$ counts how many independent copies of $X$ are in the interval $(-\infty, x],$ it has a Binomial distribution with parameters $n$ and $F(x)$ and you're done. $\endgroup$
    – whuber
    Jan 3, 2023 at 16:51

2 Answers 2

6
$\begingroup$

Define $$Y_i(x)=\mathbb I_{\{X_i\leq x\}}$$ $\forall i\in\{1, 2,\ldots, n\}.$

Notice $$Y_i(x) \overset{\text{iid}}{\sim}\mathcal{Ber}(\theta)\tag 1\label 1$$ where $\theta := F(x) . $

Now express (how?) $$n \hat F_n(x) =\sum_{i=1}^n Y_i(x) ;\tag 2$$

Use $\eqref 1$ above to yield $\operatorname{Var}(F_n(x)). $

$\endgroup$
2
  • 1
    $\begingroup$ concise and straight to the point (+1). Trivial note: the numbering (2) seems redundant, it is not actually used. $\endgroup$
    – utobi
    Jan 3, 2023 at 21:10
  • 1
    $\begingroup$ I numbered that for if OP needed, I would have expanded. $\endgroup$ Jan 3, 2023 at 22:16
6
$\begingroup$

Note that you can write $\mathop{\hat{F}_n}\left(x\right)$ as $\mathop{\hat{F}_n}\left(x\right) = \mathop{R_n}\left(x\right)/n$, where $\mathop{R_n}\left(x\right) \sim \mathop{\text{Binomial}}\left(n, \mathop{F}\left(x\right)\right)$.

Proof.

$\mathop{R_n}\left(x\right) \mathrel{:=}\sum_{i=1}^n \mathop{\mathbf{1}_{\left(-\infty,\, x\right]}}\left(X_i\right)$ counts the number of successes, meaning the number of $X_i$s in $\left(-\infty, x\right]$, in $n$ independent Bernoulli trials with success probability $\mathop{\mathbb{P}}\left(X_i \in \left(-\infty, x\right] \right) = \mathop{F}\left(x\right)$ each. Hence, $\mathop{R_n}\left(x\right) \sim \mathop{\text{Binomial}}\left(n, \mathop{F}\left(x\right)\right)$.
By definition, $\mathop{\hat{F}_n}\left(x\right) = n^{-1}\sum_{i=1}^n \mathop{\mathbf{1}_{\left(-\infty,\, x\right]}}\left(X_i\right)$ and the statement follows.

$\endgroup$
7
  • $\begingroup$ That is $R_n(x)$ here? $\endgroup$
    – CORy
    Jan 3, 2023 at 11:26
  • 1
    $\begingroup$ @CORy Hint: It's a sum of $n$ i.i.d. indicator random variables. $\endgroup$
    – statmerkur
    Jan 3, 2023 at 11:30
  • $\begingroup$ So maybe $\hat{F}_n(x) = 1/n \sum i$ where i is the random variable? $\endgroup$
    – CORy
    Jan 3, 2023 at 11:58
  • 2
    $\begingroup$ @CORy what does your random variable $i$ stand for? $\endgroup$
    – statmerkur
    Jan 3, 2023 at 12:03
  • 1
    $\begingroup$ @CORy You already use $\text{Var}(R_n(x)) = nF(x)(1 - F(x))$ in your edited derivation. What does this tell you about $\text{Var}(R_n(x)/n)$? Do you need the other variance decomposition steps? $\endgroup$
    – statmerkur
    Jan 3, 2023 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.