# Simple Random sampling with replacement - Variance

If just a single unit is drawn at random from the whole population and $$y_1$$ is the value of $$y$$ for this sampled unit, how can I prove that the $$\mathbb{V}(y_1)= (1 − 1/N)\sigma^2$$ (where $$N$$ is the population size and $$\sigma^2$$ the population variance?

To prove that the expected value of $$y_1$$ was $$\mu$$ (the population mean), I just used the definition of the expected value, as the probability of taking any y from the sample is $$1/N$$, the result came pretty fast. I don't see any way to do the same for the variance. Any tips or help toward the right idea to prove the results of the variance would be very much appreciated. Thank you in advance!

• Since by definition $\operatorname{Var}(Y_1) = \sigma^2,$ this will be impossible to prove! – whuber Oct 18 '18 at 13:39
• y1 is from the sample you sampled from the population (if that makes sense haha) σ^2 is the population variance, not the sample one. – Student number x Oct 18 '18 at 13:59
• @Student How is your $\sigma^2$ defined? Denominator is N or N-1? – user158565 Oct 18 '18 at 16:46
• This question is answered en passant in my post at stats.stackexchange.com/a/420681/919, which derives this variance formula in an analysis of stratified random sampling without replacement from a finite population. – whuber Aug 17 at 14:24

I am going to chime in with an answer to this question, because it is one where there is sometimes some confusion, owing mostly to the way in which classical sampling theory operates. Within classical sampling theory (as opposed to the super-population method), all sampling results are derived conditional on the empirical distribution of the finite population (which is equivalent to conditioning on the ordered population vector). This conditioning is usually implicit, but in this answer I will make it explicit. The other aspect of classical sampling theory that can be confusing is that the notation differs from the super-population method, and in particular, it is usual to define the "variance" parameter $$\sigma^2$$ as the population variance, without use of Bessel's correction. This is what whuber is referring to in some of his comments on your question.

Explicit framing of your problem: The first important issue to note here is that the result you are seeking to prove only holds when the "variance" you are using is the variance of the population, with adjustment for Bessel's correction. To avoid ambiguity in the notation, and consistent with sampling notation used in O'Neill (2014), I am going to denote this statistic as:

$$S_N^2 = \frac{1}{N-1} \sum_{i=1}^N (Y_i - \bar{Y}_N)^2.$$

(Note: This is the statistic you have denoted by $$\sigma^2$$. There is some ambiguity in this latter notation, since it is used to refer to different things in different types of sampling theory. To avoid ambiguity, my preference is to use $$\sigma^2$$ to refer to the variance parameter for an infinite population, and use notation like the above for a finite population.) As stated above, the variance result you are seeking is a classical sampling result that is implicitly a conditional variance, conditional on knowing the population values (but not their order). To frame this problem explicitly, we will let $$\tilde{\boldsymbol{Y}} = (Y_{(1)},...,Y_{(N)})$$ be the ordered population vector, which gives you all the population values, without telling you their order. (Note that this vector gives the same information as the empirical distribution function $$F_N$$ for the population.) The explicit form of the result you are asserting is:

$$\mathbb{V}(Y_i|\tilde{\boldsymbol{Y}}) = \frac{1}{N} \sum_{i=1}^N (Y_i - \bar{Y}_N)^2 = \Big( 1-\frac{1}{N} \Big) S_N^2.$$

This is an explicit form of the result you are trying to prove. In the above notation I have used explicit conditioning notation, which would usually be treated as implicit within classical sampling theory. This makes it clearer exactly what statistic you are looking at as the "variance", and it also makes the conditioning in the variance calculation clearer. (Incidentally, you can see from the above equation why many classical theorists take the view that the population variance should not include Bessel's correction; the variance statistic without this correction is the conditional variance of a single sample value, conditional on the population values.)

Proving the result: To prove this result, let $$K \sim \text{U} \{ 1,...,N \}$$ denote a randomly selected index for a sampled unit, and let $$I_k \equiv \mathbb{I}(K=k)$$ denote the indicator variables associated with that random index. Using these indicators, we can write the value of a randomly sampled unit as:

$$Y_K = \sum_{k=1}^N I_k Y_k.$$

It is simple to establish that $$\mathbb{V}(I_k, I_{k'}) = \tfrac{1}{N} (1-\tfrac{1}{N})$$ and $$\mathbb{C}(I_k, I_{k'}) = -\tfrac{1}{N^2}$$ for $$k \neq k'$$, so we have:

\begin{aligned} \mathbb{V}(Y_i|\tilde{\boldsymbol{Y}}) = \mathbb{V}(Y_K|\tilde{\boldsymbol{Y}}) &= \mathbb{V} \Bigg( \sum_{k=1}^N I_k Y_k \Bigg| \tilde{\boldsymbol{Y}} \Bigg) \\[6pt] &= \sum_{k=1}^N \mathbb{V} (I_k Y_k| \tilde{\boldsymbol{Y}} ) + \sum_{k \neq k'} \mathbb{C} ( I_k Y_k I_{k'} Y_{k'} | \tilde{\boldsymbol{Y}} ) \\[6pt] &= \sum_{k=1}^N Y_k^2 \cdot \mathbb{V} (I_k) + \sum_{k \neq k'} Y_k Y_{k'} \cdot \mathbb{C} ( I_k I_{k'}) \\[6pt] &= \frac{1}{N} \Big( 1-\frac{1}{N} \Big) \sum_{k=1}^N Y_k^2 - \frac{1}{N^2} \sum_{k \neq k'} Y_k Y_{k'} \\[6pt] &= \frac{1}{N} \Bigg[ \Big( 1-\frac{1}{N} \Big) \sum_{k=1}^N Y_k^2 - \frac{1}{N} \sum_{k \neq k'} Y_k Y_{k'} \Bigg] \\[6pt] &= \frac{1}{N} \Bigg[ \sum_{k=1}^N Y_k^2 - \frac{1}{N} \sum_{k =1}^N \sum_{k' =1}^N Y_k Y_{k'} \Bigg] \\[6pt] &= \frac{1}{N} \Bigg[ \sum_{k=1}^N \Big( Y_k^2 - Y_k \frac{1}{N} \sum_{k' =1}^N Y_{k'} \Big) \Bigg] \\[6pt] &= \frac{1}{N} \Bigg[ \sum_{k=1}^N Y_k^2 - N \bar{Y}_N^2 \Bigg] \\[6pt] &= \frac{1}{N}\sum_{k=1}^N (Y_k - \bar{Y}_N)^2 \\[6pt] &= \Big( 1 - \frac{1}{N} \Big) S_N^2. \\[6pt] \end{aligned}

(Note that this is the same proof method as the other answer by user158565, but I will add some additional steps to fill in some blanks.)

When the sample size = 1, with or without replacement does not matter.

All the summation is from 1 to $$N$$.

For finite population, the variance is defined as: $$\sigma^2 = \frac 1 {N-1} \sum(Y_i-\bar Y)^2$$ where $$N$$ is population size.

Let $$Z$$ be the value you get from sample with sample size 1.Then $$Z = \sum Z_iY_i$$ where $$Z_i$$ is the random variable, = 1 if $$Y_i$$ is sampled, and =0 if not selected.

When sample size = 1, we have,

$$Var(Z_i) = \frac 1N \left(\frac{N-1}N\right)$$ $$Cov(Z_i,Z_j) = E(Z_iZ_j)-E(Z_i)E(Z_j) = 0 - \frac 1N \frac 1N = -\frac 1{N^2}$$

We have $$Var(Z) = \sum Var(Z_i) Y_i^2 + \sum_{i\ne j}Cov(Z_i,Z_j) Y_iY_j$$ $$= \frac {N-1}{N^2}\sum Y_i^2 -\frac1{N^2}\sum_{i\ne j} Y_iY_j$$ $$=\left(1-\frac 1N\right)\left( \frac 1 {N-1} \sum (Y_i-\bar Y)^2\right)$$ $$= \left(1-\frac 1N\right) \sigma^2$$

• How do you compute a sample SD with a "single unit" in the sample? For a finite population the variance is definitely not defined as you give it here: the denominator is $N,$ not $N-1.$ One way to see this is to consider a finite population of two values, $1$ and $0$. In this case the random variable is Bernoulli$(1/2),$ but your formula with $N-1$ gives $1/2$ for its variance rather than $1/4$ as it should be. – whuber Oct 18 '18 at 15:40
• @whuber Why need a sample SD? About the definition of variance see page 2 on support.sas.com/rnd/app/stat/examples/SurveyVariance/… – user158565 Oct 18 '18 at 15:58
• I wonder what SAS's reply to my question about the Bernoulli variance would be? I suspect their manual is referring to calculations related to sampling from finite populations without replacement. As you have noted at the beginning of your reply, this is not a sample without replacement. Unless you're willing to live with a stark mathematical inconsistency, something has to give--and I would like to suggest that changing the definition of variance is not the right way to do it. – whuber Oct 18 '18 at 16:02
• As I said in Answer, when the sample size = 1, there is no difference between with and without replacement. They DEFINE the variance with N-1 as denominator for variance. Maybe we should ask "Student number x" what his $\sigma^2$ mean. – user158565 Oct 18 '18 at 16:13
• My point is that this definition is not relevant here--and in fact gives the wrong results. Another way to see this is to compare the statistical characteristics of $Y$ drawn from the Bernoulli population I previously described, with $N=2,$ and of $Y^\prime$ drawn from a population of $N=4$ where for two of them $Y^\prime=0$ and for the other two $Y^\prime=1.$ Obviously $Y$ and $Y^\prime$ have identical characteristics, so I ask you this: which value of $N$ will you use for computing their (common) dispersion: $N=2$ or $N=4$?? – whuber Oct 18 '18 at 17:14