Sampling probability distribution-how many samples?

Question

Simple question.

I have a rv $X$ whose mean and variance I know but am interested in the true distribution, $P(X \le a)$ which I do not know. So I resort to sampling $X$. I do this $N$ times and create a approximate pmf for $X$. I want to know how "good" this is. How do I quantify this?

Okay, more realistically I am interested in this specific value $P(X \le k)$. Say the true answer to this is $\alpha$ which I do not know. But using my $N$ samples I get an answer to this as $\beta$. Now how close is $\alpha$ to $\beta$ probabilistically?

Answer 1

The Central Limit Theorem has something to say about your last question. Specifically, the usual way of estimating such as $P\left(X \leq k \right)$ consists of using the empirical distribution function (ECDF) and computing

$$F_n(k) = \sum_{i=1}^n \frac{1}{n} \mathbf{1} \left(X_i \leq k \right)$$

where we just count the number of observations $X_1, \ldots, X_n$ that fall short of $k$ and divide by the total number observations. That is, our most intuitive estimator for this probability is the corresponding sample proportion.

It turns out that this estimator has fantastic statistical properties. It can be shown that it converges, as $n \to \infty$, to $P\left(X\leq k \right)$ with probability one and that it does so uniformly for all values of $k$. Since $F_n(k)$ is also a sum of iid terms, one can further apply the CLT to make probabilistic statements about the distance from the true number. It's not very difficult to show that

$$F_n(k) \xrightarrow{D} \mathcal{N} \left( P(X\leq k), \frac{ P(X\leq k)\left(1-P(X\leq k\right)} {n} \right) $$

By using the properties of the Normal distribution you can make probabilistic statements about the quantity $|F_n(k)-P(X\leq k)|$ after plugging in $F_n(k)$ in the variance of the asymptotic distribution.

Answer 2

As noted in earlier answers, the empirical frequency$$\frac{1}{N}\sum_{i=1}^N \mathbb{I}_{(-\infty,x]}(X_i) \stackrel{\Delta}{=} \frac{1}{N} S_N$$is associated with a Binomial variate $S_N\sim\text{Bin}(N,p)$ for which a conservative confidence interval can be constructed. There are many proposals for this construction. Including one by George Casella.

For instance, Jeffreys' confidence interval on $p$ with coverage $1-2\alpha$ is given by

N=50;SN=13;alpha=.05
qbeta(alpha,.5+SN,.5+N-SN)
[1] 0.1268967
qbeta(1-alpha,.5+SN,.5+N-SN)
[1] 0.4364774

Answer 3

If it's just P(X≤k) you need for a given k, then it becomes a relatively simple problem. You can treat the problem as sampling from a binomial random variable Y~ Bin(N,p) where you are interested in estimating the probability of success p (or what you call α). For each sampled value x out of the distribution of X, a success will be x≤k and the estimate for p (or α) will be y/N where y is the number of successes (number of sampled x's which are ≤k). There is no definite answer to how close the estimate is, it's up to you to determine. You can either construct a confidence interval for p (with, say, 0.95 confidence) and see if the interval is narrow enough for you. Conversely, you can decide what the margin of error for p you are allowing is (for example +-0.02), and determine if the confidence level obtained is high enough. A third way is to to determine both the margin of error and the confidence level and solve for the sample size N. I just noticed that the title of your question is about the needed sample size - this does not appear in the question itself. For determining N you need to have both a margin of error (E) and a confidence level CL (which determines Z as the value of the standard Normal distribution with probability (1-CL)/2 to the right of it) and then N=0.25(Z/E)² The 0.25 is conservative - assuming you don't know anything about p (or α).