22
$\begingroup$

I have been facing hard time understanding meaning of "random sample" as well as "iid random variable". I tried to find out the meaning from several sources, but just got more and more confused. I am posting here what I tried and got to know:

Degroot's Probability & Statistics says:

Random Samples / i.i.d. / Sample Size : Consider a given probability distribution on the real line that can be represented by either a p.f. or a p.d.f. $f$. It is said that $n$ random variables $X_1 , . . . , X_n$ form a random sample from this distribution if these random variables are independent and the marginal p.f. or p.d.f. of each of them is $f$. Such random variables are also said to be independent and identically distributed, abbreviated i.i.d. We refer to the number n of random variables as the sample size.

But one of the other statistics book I have says:

In a Random Sampling, we guarantee that every individual unit in the population gets an equal chance(probability) of being selected.

So, I have a feeling that i.i.d.s are elements that construct random sample, and the procedure to have random sample is random sampling. Am I right?

P.S.: I am very confused about this topic, so I will appreciate elaborate reply. Thanks.

$\endgroup$
4
  • 7
    $\begingroup$ The independence part is very important because we can have a sample in which all the variables are identically distributed (have the same marginal distribution) but are not independent. Such a sample can still be considered as a random sample but not of the experiment that you think it is a random sample of. See this question. $\endgroup$ Commented May 18, 2014 at 15:08
  • $\begingroup$ The question does not seem to make a statistical sense. The iid and random sample are clearly distinct concepts established by literates. $\endgroup$
    – user10619
    Commented May 19, 2014 at 15:53
  • 3
    $\begingroup$ @subhashc.davar Are they? According to one definition: "A random sample is a sequence of independent, identically distributed (IID) random variables". So it seems iid and random sample are the same thing? The cited paragraph in Degroot's Probability & Statistics basically says the same. I find it confusing because a "sample" is sometimes an individual or a set of individuals, and sometimes a sequence of random variables. $\endgroup$
    – Gary Chang
    Commented Dec 19, 2014 at 10:58
  • $\begingroup$ @Gary Chang The definition you cited pertains to pdf. The sample of random variables has been popular in the discipline of psychometry. Generally, it is used with reference to reliability Or validity estimate and for a factor analysis. The psychometry are interested in establishing equivalence of tests for a domain. The iid concept appears to originate from linear algebra. A sample could be from a given population of individuals and/ or from a population of (random) variables depending on the purpose of a study. The present day statistics appears to have borrowed from the measurementtheory. $\endgroup$
    – user10619
    Commented Dec 19, 2014 at 13:10

4 Answers 4

12
$\begingroup$

You don't say what the other statistics book is, but I'd guess that it is a book (or section) about finite population sampling.

When you sample random variables, i.e. when you consider a set $X_1,\dots,X_n$ of $n$ random variables, you know that if they are independent, $f(x_1,\dots,x_n)=f(x_1)\cdots f(x_n)$, and identically distributed, in particular $E(X_i)=\mu$ and $\text{Var}(X_i)=\sigma^2$ for all $i$, then: $$\overline{X}=\frac{\sum_i X_i}{n},\quad E(\overline{X})=\mu,\quad \text{Var}(\overline{X})=\frac{\sigma^2}{n}$$ where $\sigma^2$ is the second central moment.

Sampling a finite population is somewhat different. If the population is of size $N$, in sampling without replacement there are $\binom{N}{n}$ possible samples $s_i$ of size $n$ and they are equiprobable: $$p(s_i)=\frac{1}{\binom{N}{n}}\quad\forall i=1,\dots,\binom{N}{n}$$ For example, if $N=5$ and $n=3$, the sample space is $\{s_1,\dots,s_{10}\}$ and the possible samples are: $$\begin{gather}s_1=\{1,2,3\},s_2=\{1,2,4\},s_3=\{1,2,5\},s_4=\{1,3,4\},s_5=\{1,3,5\},\\ s_6=\{1,4,5\},s_7=\{2,3,4\},s_8=\{2,3,5\},s_9=\{2,4,5\},s_{10}=\{3,4,5\}\end{gather}$$ If you count the number of occurrences of each individual, you can see that they are six, i.e. each individual has an equal chance of being selected (6/10). So each $s_i$ is a random sample according to the second definition. Roughly, it is not an i.i.d. random sample because individuals are not random variables: you can consistently estimate $E[X]$ by a sample mean but will never know its exact value, but you can know the exact population mean if $n=N$ (let me repeat: roughly.)${}^1$

Let $\mu$ be some population mean (mean height, mean income, ...). When $n<N$ you can estimate $\mu$ like in random variable sampling: $$\overline{y}_s=\sum_{i=1}^n y_i,\quad E(\overline{y}_s)=\mu$$ but the sample mean variance is different: $$\text{Var}(\overline{y}_s)=\frac{\tilde\sigma^2}{n}\left(1-\frac{n}{N}\right)$$ where $\tilde\sigma^2$ is the population quasi-variance: $\frac{\sum_{i=1}^N(y_i-\overline{y})^2}{N-1}$. Factor $(1-n/N)$ is usally called "finite population correction factor".

This is a quick example of how a (random variable) i.i.d. random sample and a (finite population) random sample may differ. Statistical inference is mainly about random variable sampling, sampling theory is about finite population sampling.


${}^1$ Say you are manufacturing light bulbs and wish to know their average life span. Your "population" is just a theoretical or virtual one, at least if you keep manufacturing light bulbs. So you have to model a data generation process and interpret a set of light bulbs as a (random variable) sample. Say now that you find a box of 1000 light bulbs and wish to know their average life span. You can select a small set of light bulbs (a finite population sample), but you could select all of them. If you select a small sample, this doesn't transform light bulbs into random variables: the random variable is generated by you, as the choice between "all" and "a small set" is up to you. However, when a finite population is very large (say your country population), when choosing "all" is not viable, the second situation is better handled as the first one.

$\endgroup$
20
  • 1
    $\begingroup$ What do you mean "individuals are not random variables?" Whuber has some really nice answers here and here which use finite population sampling to explain the concept of a random variable. $\endgroup$
    – jsk
    Commented May 18, 2014 at 17:54
  • $\begingroup$ I mean what I've said: if $n=N$ then there is no uncertainty. $\endgroup$
    – Sergio
    Commented May 18, 2014 at 18:15
  • $\begingroup$ That was not helpful in clarifying your statement that appears to be in direct contradiction with those made in the links. Please, there is no need to be defensive. The point about $n=N$ has nothing to do with the statement I'm curious about. Besides, is a degenerate random variable not a random variable? $\endgroup$
    – jsk
    Commented May 18, 2014 at 19:06
  • $\begingroup$ Defensive? You didn't understand those links. As whubner says, a) the tickets-in-a-box model is just a toy example to avoid "this is graduate level stuff" complains; b) he avoids calling "population" the tickets in a box, and explains why. So there is no contradiction. If one can understand what whubner has said. BTW, I'm not a random variable, are you? $\endgroup$
    – Sergio
    Commented May 18, 2014 at 19:28
  • $\begingroup$ IMHO, of course. $\endgroup$
    – Sergio
    Commented May 18, 2014 at 19:44
6
$\begingroup$

I will not bore you with probabilistic definitions and formulas, which you may easily pick up at any textbook (or here is a good place to start)

Just think of this intuitively, random sample is a set of random values. In general, each one of the values may either be identically or differently distributed. $i.i.d.$ sample is a special case of random sample, such that every value comes from the same distribution as the others and its value does not have any influence upon other values. Independence deals with $how$ the values were generated

$i.i.d$ example: draw a random card from a deck and return it back (do this 5 times). You will get 5 realized values (cards). Each one of these values comes from a uniform distribution (there is equal probability to get each one of the outcomes) and each draw is independent of the others (i.e. the fact that you get an ace of spades in the first draw, does not influence in any way the result you may get in other draws).

non $i.i.d.$ example: Now do the same thing, but without returning the card to the deck (I hope you feel the difference by now). Again you will have 5 realized values (cards) after you do this. But clearly they are dependent (the fact that you draw the ace of spades on the first draw, means you will not have a chance to get in on the 2nd draw).

$\endgroup$
0
1
$\begingroup$

A Random Variable usually written X, is a variable whose possible values are numerical outcomes of a random phenomenon. The random phenomenon may produce outcomes that have numerical values captured by the random variable --e.g. number of heads in 10 tosses of a coin or incomes/heights etc in a sample -- but that is not necessary .
More generally a Random Variable is a function that maps random outcomes to numeric values. E.g. each day may be sunny, cloudy or rainy. We can define a Random Variable that takes the value 1 if it is rainy, 2 if it is cloudy and 3 if it is sunny. The domain of a random variable is the set of possible outcomes.
To establish a Random Variable there must be a process or experiment that is associated with possible outcomes that can not be predicted with certainty.

Coming now to the issue of independence. Two Random Variables are independent if the value of one of them does not affect the PDF of the other. We don't revise our predictions regarding the probabilities of different values of one variable when we know something about the other variable. Therefore in the case of independence the Posterior PDFs are identical to the Prior PDFs. E.g. when we toss a unbiased coin repeatedly, the information we have about the outcome of the 5 prior tosses does not affect our prediction about the current toss, it will be always 0.5. However, if the bias of the coin is unknown and is modeled as a Random Variable, then the outcome of the previous 5 tosses affects our predictions regarding the current toss because it allows us to make inferences regarding the unknown bias of the coin. In that case the Random Variables capturing the number of Heads in a sequence of n tosses are dependent and not independent.

Coming now to the issue of Sampling. The purpose of Sampling is to inform us about the properties of an underlying distribution that is not known and must be inferred. Remember that a Distribution refers to the relative likelihood of possible outcomes in the Sample Space (which may also be a Conditional Universe). So when we Sample we chose a finite number of outcomes from the Sample space and we reproduce the Sample Space in a smaller more manageable scale. Equal probability then refers to the process of the Sampling not the probability of the Outcomes in the Sample. Equal probability sampling implies that the Sample will reflect the proportions of the outcomes in the original Sample Space. E.g. if we ask 10,000 people if they have ever been arrested it is probable that the sample we will end up will not be representative of the Population -- the Sample Space-- since people who would have been arrested might refuse to reply, therefore the proportion of possible outcomes (arrested - not arrested) will differ between our sample and the population for systematic reasons. Or if we chose a particular neighborhood to conduct a survey the results will not be representative of the City as a whole. So equal probability sampling implies that there are no systematic reasons --other than pure randomness-- that makes us believe that the proportions of possible outcomes in our sample are different from the proportions of outcomes in the Population / Sample Space.

$\endgroup$
-1
$\begingroup$

A random sample is a realization of a sequence of random variables. Those random variables may be i.i.d or not.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.