1
$\begingroup$

Does the definition of a sampling distribution always assume our (theoretical) ability to generate i.i.d samples of the underlying RVs (together, as a collection of RVs) forming the statistic?

From Wikipedia:

The sampling distribution of a statistic is the distribution of that statistic, considered as a random variable, when derived from a random sample of size $n$

I assume the answer is an implicit yes because by definition all samples from the distribution of the statistic would be identically distributed, and thus, each such sample of the statistic would come from an independent and identically distributed sample of the underlying random variables.

However, the article makes no mention of i.i.d. so it makes me doubt.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

No, the answer is "It depends".

If the experiment consists of choosing $k$ balls without replacement from a set of $n$ balls numbered from $1$ to $n$, then the outcome (or sample values if you prefer) must be a set of distinct numbers from $\{1,2,\cdots, n\}$ and in this case, the sample values are not independent.

If the experiment consists of choosing $k$ balls with replacement from a set of $n$ balls numbered from $1$ to $n$, then the outcome (or sample values if you prefer) can repeat (that is, they need not be $k$f distinct numbers from $\{1,2,\cdots, n\}$. In this case, the sample values are independent.

Curiously enough, in both these examples, the sample values are nonetheless identically distributed, that is, "i.i.d." should not be an indivisible mantra to you: things can be identically distributed but not independent, or independent but not identically distributed.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks Dilip. But doesn't each sample of the statistic correspond to a run of that experiment? So while the samples collected within an experiment may not be independent, the samples across experiments are iid. For example, if the statistic is the sample (ball index) mean for each experiment of drawing K balls (e.g. say avg of choosing ball 2 and then ball 4 = (2+4)/2 = 3), we can assume that the k-long sequence of ball indices (e.g. the sequence of random variables as a random variable itself, which is different from the sample mean), is iid across samples of the statistic. Right? $\endgroup$
    – Josh
    Jul 25, 2020 at 20:26
  • 1
    $\begingroup$ @Josh To me, sampling distribution means the distribution of the random variables whose present values (realizations) are the sample values observed. So, in my examples, $X_1, X_2, \cdots, X_k$ are discrete random variables taking on values in the set $\{1,2,\cdots, n\}$ and these are identically distributed in both cases, but dependent in one case and independent in the other. If for you, the statistic is the mean value of the $k$ samples, then the question of i.i.d. does not arise at all: with just one value, there is nothing to be independent of and no other distribution $\endgroup$
    – Dilip Sarwate
    Jul 25, 2020 at 20:38
  • $\begingroup$ ---- to be identically distributed with. And yes, if you repeat the experiment, and the repetition is independent of the first experiment, then the distribution of the sample mean is the same in both experiments and the independence of the two trials of the experiment is by assumption. But, if $k < n/2$ and the second experiment draws $k$ balls from the depleted urn (with $n-k$ balls left in it) at the end of the first experiment, then the experiments are not independent and the mean of the second trial is not independent of the mean of the first trial at all. $\endgroup$
    – Dilip Sarwate
    Jul 25, 2020 at 20:44
  • $\begingroup$ Perhaps the crux of the Q is the following: I think of a statistic $S$ as $S = f(\mathbf{X})$, where $f$ is a function defined over $\mathbf{X} = (X_1, \ldots, X_n)$. Since $X_i$ is a RV, I consider $\mathbf{X}$ a RV itself. If I understand you correctly, you are saying that $X_i$ and $X_j$ need not be i.i.d, and you don't think of $\mathbf{X}$ as a RV (or one worth discussing). If you did, however, you would grant that $\mathbf{X}_k$ and $\mathbf{X}_l$ (indexing over samples of the vector $\mathbf{X}$) are i.i.d in the context of the sampling distribution of $S$, correct? $\endgroup$
    – Josh
    Jul 26, 2020 at 16:14
1
$\begingroup$

Sampling from finite populations

The sampling theory for finite populations is usually applied to objects of a non-random, determined nature, for example all adults aged 18 or older living in a country at a given date.

All is based on the probability $\pi_i$ that element $i$, $i=1,\dots,N$, is selected for the sample, and the probability $\pi_{ij}$ that elements $i$ and $j$ are jointly included in the sample.

There are statistics and sampling distributions, but not i.i.d. samples because the reciprocal of the selection probability, $w_i=\pi_i^{-1}$, is a random variable, but $y_i$, the individual values of a variable of interest are considered fixed (see here.) In other words, in sampling from finite populations a random sample is not a set of i.i.d. random variables, the meaning of "random" and "randomization" is not the same as in sampling from infinite populations, what is (or is not) random is elements selection.

Sampling from infinite populations

The sampling theory for infinite populations was originally applied to the measurement of astronomical and geodesical quantities, which involved continuously-distributed random errors. The measurements formed a sample from an infinite set of possible results, and the observational errors were subject to a probability distribution (triangular according to Simpson, double exponential according to Laplace, then gaussian). The results of the observations were treated as experimentally found values of the random variables subject to this distribution; this is how satistical inference was born (see Stigler, The History of Statistics, and here.)

In sampling from infinite populations (i.e. from random variables) a sample is a set of random variables, your observations are realizations of those random variables. You need their joint distribution, which is easily handled if you may assume independence and identical distribution.

This is why, in sampling from random variables, you try to select observations that can be viewed as realizations of i.i.d. random variables (so called "randomization"). In cross-sectional data, data collected by observing many subjects at the one point or period of time, this might be hard, but is often feasible.

However, assuming i.i.d. in time-series data would be nice, but may be impossible, because a time series of a random variable has often serial dependence.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.