3
$\begingroup$

Background:

The weak law of large numbers states that for a sequence $X_1,X_2,\ldots,X_n$ of iid RVs, with expectation $\mu$ and variance $\sigma^2$, the sample mean converges to $\mu$:

$$\hat{X}=\frac{\sum_{i=1}^nX_i}{n}\stackrel{p}{\rightarrow}\mu$$

That is the sample mean converges in probability to the population mean as the number of RVs approaches $\infty$.

Question:

How can you obtain infinitely many iid random variables from a finite population? How do you in practise check that your random variables are independent?

Edit:

By finite population I mean that you consider a population of individuals. This population is finite. You consider a characteristic in the population. You model the characteristic with a random variable. I do not mean that the range of the random variable is finite.

Edit 2

We know that $\mu$ is a population characteristic. Let us assume the population is of size $n$. Denote by $Y$ the random variable that describes the population characteristic. Then $\mu=\text{E}(Y)$. Let $Y_1,\ldots,Y_n$ denote the random variables of respectively individual 1 to $n$. By definition $\mu=\frac{\sum_{i=1}^nY_i}{n}$. We then make a sample from the population. How can we obtain a sample of size $n+1$ or $n\rightarrow\infty$ that is iid from a population that is finite? Here some say that we can sample from $Y_1,\ldots,Y_n$ WITH replacement.

Edit 3

If we consider a sample of size $N$ with $N\leq n$, where the sampling is done without replacement, can we can obtain a iid sample if the original RVs are independent? If $Y_1,\ldots\ Y_n$ are iid and we let $X_1=Y_1,\ldots, X_n=Y_n$ and consider any subset of $X_1,\ldots, X_n$, then this subset will consist of iid RVs, right? Am I missing some point here?

Improve this question
$\endgroup$
14
  • 3
    $\begingroup$ So you don't ever have infinitely many. It's that after you have a large finite number, you get `close' to the limit value of $\mu$. $\endgroup$
    – VCG
    Commented Aug 25, 2016 at 12:39
  • $\begingroup$ So, if the random variable's range is not finite, it clearly models some other variability in addition to that of random sampling the individuals from the finite set of individuals. But in this case you will not find out $E[X_i]$ exactly even by sampling all individuals in the finite set. Can you clarify whether you are looking for estimating $E[X_i]$ or $(X_1+\ldots+X_N)/N$? In the first case, no, you cannot physically obtain more samples than there are individuals; in the second case I don't see how LLN is supposed to be related. $\endgroup$
    – Juho Kokkala
    Commented Aug 26, 2016 at 9:33
  • 1
    $\begingroup$ It is not possible for the $Y_i$s be iid and $E(Y_1)=\mu=\frac{\sum_{i=1}^n Y_i}{n}$ to hold (except technically iid holds if $Y_i$ is a constant rv). If for example $Y_i \sim N(10,5)$, the mean of any finite number of $Y_i$s shall not be exactly $10$ (except with probability $0$). So, $\mu$ cannot be both the mean of the underlying distribution and the mean of the $n$ iid samples. $\endgroup$
    – Juho Kokkala
    Commented Aug 27, 2016 at 15:11
  • 1
    $\begingroup$ You are still mixing the distribution of the underlying random variable and the distribution observed in your sample(the finite population) -- I don't know what it means to say that 'the distribution of $Y$ is truly a normal distribution'. I'd also avoid the term 'population mean' since it is often used to refer to the model parameter (mean of a hypothetical infinite population) -- if you model the $Y_i$s as iid realizations from some distribution, then in modeling terms your finite population is really a sample even if it happens to contain every entity that exist in the real world. $\endgroup$
    – Juho Kokkala
    Commented Aug 27, 2016 at 18:45
  • 1
    $\begingroup$ This comment chain is getting all too long, and the issue does not seem to be very related to the law of large numbers (I suspect that if the unclarities related to finite populations, samples, etc. are resolved, it will turn out that the question as formulated is not very relevant). Perhaps you want to post a new question. If you have a real-life application in mind, it might be helpful to describe that, too, as that may make it easier to see behind the formalizations and possible errors in them. $\endgroup$
    – Juho Kokkala
    Commented Aug 27, 2016 at 18:50

1 Answer 1

Reset to default
2
$\begingroup$

So if your question is "how can we apply the LLN with finite data" then I give the answer I gave in the comments:

"So you don't ever have infinitely many. It's that after you have a large finite number, you get `close' to the limit value of $\mu$"

But, if your question is rather: How do we reconcile the LLN with a finite population, then I lead to to this fantastic post about finite population sampling :Are "random sample" and "iid random variable" synonyms?

The other post about $n\to N$ being equivalent to $n\to\infty$ is problematic as LLN is a result of probability limits and sequences and those necessarily are not finite in index.

So n is only ever able to reach N if we sample without replacement, meaning the samples are no longer iid. If we sample with replacement then we never run out of n.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your answer, @VCG, really appreciated. So you have given two answers. In answer (1) you say that you really do not ever have an infinite sequence of i.i.d RVs, but as the number of RV gets big enough, you get close enough to the limit $\mu$? $\endgroup$
    – FredrikAa
    Commented Aug 26, 2016 at 19:22
  • $\begingroup$ Hmm..I'm trying to understand your question and your edits. We're getting into heavy terminology territory so I'll think more about it. $\endgroup$
    – VCG
    Commented Aug 26, 2016 at 20:31

Not the answer you're looking for? Browse other questions tagged or ask your own question.