Prove that the variance of the sample mean is smaller than that of the mean of a simple random sample of the same size n drawn with replacement

Question

In statistics, a simple random sample is a subset of individuals chosen (one by one) from a population. Each individual is chosen randomly such that each individual has the same probability of being chosen at any stage during the sampling process, and each subset of k individuals has the same probability of being chosen for the sample as any other subset of k individuals. From a population of size N with finite variance, a simple random sample of size n is drawn without replacement, and a real-valued characteristic X measured to yield observations Xj(j =1,..., n).

Prove that the expected squared error of the sample mean, i.e. the variance of the sample mean, is smaller than that of the mean of a simple random sample of the same size n drawn with replacement.

The question came from the book "A Course in Mathematical Statistics and Large Sample Theory" by Rabi Bhattacharya, Lizhen Lin, and Victor Patrangenaru.

Answer 1

For self-study questions like this the CV policy is to give hints rather than complete answers.

Here the question focuses on something that is special to finite populations, and it seems to me to be a graduate level question rather than an elementary question. Here is a hint. When you select observations with replacement, all the observations are independent and from the same population, so you can apply the usual rules for the variance of a sum of iid random variables, which leads to the usual well known rule for the variance of a sample mean. Can you see why they are independent?

When you select observations without replacement, they are not independent because each observation is exclusive of the others. In fact, all the values in the SRS are negatively correlated which each other, and this reduces the variance of their mean. You have to figure out why they are negatively correlated. I assume that there must be clues in the textbook, and most likely in the same chapter that you took the question from.

Here is another clue. Suppose you draw a SRS without replacement of size $N$ (same size as the population). What would the mean be? If you did this more than once, would you always get the same value? Hence, what would the variance be? Why does this imply negative correlation?

If all $X_i$ and $X_j$ share the same correlation for $i\ne j$, what is the variance of $\bar X$?

As a final hint, you would find it helpful to introduce some notation to clarify your thoughts, e.g., let $\mu$ and $\sigma^2$ be the population mean and variance.