Simple question.

I have a rv $X$ whose mean and variance I know but am interested in the true distribution, $P(X \le a)$ which I do not know. So I resort to sampling $X$. I do this $N$ times and create a approximate pmf for $X$. I want to know how "good" this is. How do I quantify this?

Okay, more realistically I am interested in this specific value $P(X \le k)$. Say the true answer to this is $\alpha$ which I do not know. But using my $N$ samples I get an answer to this as $\beta$. Now how close is $\alpha$ to $\beta$ probabilistically?

  • 2
    $\begingroup$ If you are only interested in $\alpha$ then you are sampling from a Bernoulli distribution with probability of success $\alpha$. You can construct a confidence interval for $\alpha$ $\endgroup$ – Hugh Mar 3 '17 at 15:14

The Central Limit Theorem has something to say about your last question. Specifically, the usual way of estimating such as $P\left(X \leq k \right)$ consists of using the empirical distribution function (ECDF) and computing

$$F_n(k) = \sum_{i=1}^n \frac{1}{n} \mathbf{1} \left(X_i \leq k \right)$$

where we just count the number of observations $X_1, \ldots, X_n$ that fall short of $k$ and divide by the total number observations. That is, our most intuitive estimator for this probability is the corresponding sample proportion.

It turns out that this estimator has fantastic statistical properties. It can be shown that it converges, as $n \to \infty$, to $P\left(X\leq k \right)$ with probability one and that it does so uniformly for all values of $k$. Since $F_n(k)$ is also a sum of iid terms, one can further apply the CLT to make probabilistic statements about the distance from the true number. It's not very difficult to show that

$$F_n(k) \xrightarrow{D} \mathcal{N} \left( P(X\leq k), \frac{ P(X\leq k)\left(1-P(X\leq k\right)} {n} \right) $$

By using the properties of the Normal distribution you can make probabilistic statements about the quantity $|F_n(k)-P(X\leq k)|$ after plugging in $F_n(k)$ in the variance of the asymptotic distribution.

  • $\begingroup$ Asymptotically, yes, but it's the rate of convergence that's essential to the question, no? $\endgroup$ – A. Webb Mar 3 '17 at 15:25
  • $\begingroup$ @A.Webb The OP is interested in making probabilistic statements, the classical CLT is all that he needs for this. $\endgroup$ – JohnK Mar 3 '17 at 15:26
  • $\begingroup$ @JohnK how do I get the mean and variance of this normal distribution as I do not know the value $P(X \le k)$? $\endgroup$ – bissi Mar 5 '17 at 15:23
  • $\begingroup$ @bissi For the variance you use the estimate that you have for $P(X\leq k )$, namely $F_n(k)$. It can be shown that the asymptotic normal distribution still holds after you plug-in the estimate. $F_n(k)-P(X\leq k)$ has by definition mean zero, so you don't have to worry about that. Just find the density of the the absolute value of a normal variable. $\endgroup$ – JohnK Mar 5 '17 at 19:37
  • 1
    $\begingroup$ @bissi No, the mean of this absolute value is not zero. The variable $F_n(k)-P(X\leq k)$ (without the absolute value) is normally distributed with mean zero and the variance you say. Based on this distribution you can compute the probability that this difference is, say less than 0.5 by $$ P\left( |F_n(k)-P(X\leq k)| \leq 0.5 \right) = P\left( -0.5 \leq F_n(k)-P(X\leq k) \leq 0.5 \right) $$ using the normal distribution. This is what you had in mind, right? $\endgroup$ – JohnK Mar 6 '17 at 10:06

As noted in earlier answers, the empirical frequency$$\frac{1}{N}\sum_{i=1}^N \mathbb{I}_{(-\infty,x]}(X_i) \stackrel{\Delta}{=} \frac{1}{N} S_N$$is associated with a Binomial variate $S_N\sim\text{Bin}(N,p)$ for which a conservative confidence interval can be constructed. There are many proposals for this construction. Including one by George Casella.

For instance, Jeffreys' confidence interval on $p$ with coverage $1-2\alpha$ is given by

[1] 0.1268967
[1] 0.4364774
  • $\begingroup$ It's interesting that your sample CDF is left-continuous. $\endgroup$ – JohnK Mar 6 '17 at 19:48
  • 2
    $\begingroup$ @JohnK: pardon my French, I often confuse ) with ] given that in French open intervals are denoted as ]a,b[. $\endgroup$ – Xi'an Mar 6 '17 at 20:50

If it's just P(X≤k) you need for a given k, then it becomes a relatively simple problem. You can treat the problem as sampling from a binomial random variable Y~ Bin(N,p) where you are interested in estimating the probability of success p (or what you call α). For each sampled value x out of the distribution of X, a success will be x≤k and the estimate for p (or α) will be y/N where y is the number of successes (number of sampled x's which are ≤k). There is no definite answer to how close the estimate is, it's up to you to determine. You can either construct a confidence interval for p (with, say, 0.95 confidence) and see if the interval is narrow enough for you. Conversely, you can decide what the margin of error for p you are allowing is (for example +-0.02), and determine if the confidence level obtained is high enough. A third way is to to determine both the margin of error and the confidence level and solve for the sample size N. I just noticed that the title of your question is about the needed sample size - this does not appear in the question itself. For determining N you need to have both a margin of error (E) and a confidence level CL (which determines Z as the value of the standard Normal distribution with probability (1-CL)/2 to the right of it) and then N=0.25(Z/E)² The 0.25 is conservative - assuming you don't know anything about p (or α).

  • $\begingroup$ I understand everything up till $y/N$ being an estimate for the quantity I am seeking. Now the question is what is the CI of this$y/N$ given the number of samples I have. So your "third way" This I don't understand from your answer.. What is say the 95%CI around my estimate as a function of N? $\endgroup$ – bissi Mar 5 '17 at 15:15
  • $\begingroup$ The first article that came up in a Google search is dummies.com/education/math/statistics/… - I hope you can follow it. $\endgroup$ – Zahava Kor Mar 7 '17 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.