6
$\begingroup$

Suppose I have $N$ independent random variables $X_n$. I draw a sample of predetermined size $K_n$ from each of them. Denote the average of each sample $\bar{\hat{X}}_n$, and the total number of observations $M = \sum_{n=1}^{N} K_n$.

Next, I use all the $M$ sample observations $\hat{X}_{n,k}$, with no regard as to which r.v. they came from, to construct the empirical distribution $\hat{F}$.

Next, I create $N$ new i.i.d. random variables $Y_n$ distributed as $\hat{F}$, and draw from the them i.i.d. samples of the same sizes $K_n$. Denote the average of each sample $\bar{\hat{Y}}_n$.

Let $T$ be some commonly used sample statistic that represents the dispersion; e.g., variance (or interquartile distance).

Finally, I compare $T_X = E[T(\bar{\hat{X}}_1, …, \bar{\hat{X}}_N)]$ and $T_Y = E[T(\bar{\hat{Y}}_1, …, \bar{\hat{Y}}_N)]$.

Under what (preferably intuitive) conditions would I find $T_X = T_Y$, $T_X > T_Y$, or $T_X < T_Y$?

Motivation:

I've seen bootstrap used in social sciences to analyze a large group of independent random variables. Specifically, the researcher would examine the variation in the sample statistics across these variables. If it seems that the variation is bigger than "expected", the researcher would draw some conclusions about the original r.v. But I could never figure out precisely what null hypothesis this approach tests, so I'm also unsure about what conclusions would be justified.

$\endgroup$
  • $\begingroup$ This is an interesting question, but as posed is quite broad. I think it will be asking quite a lot to obtain a concise answer which addresses your question as currently stated. Is there a particular form of the problem that is especially interesting to you? Also, are you assuming that the $\{X_n\}$ are independent random variables? That wasn't clear from your question. My suggestion: Reformulate the question to handle as concrete of an instance as possible. This will allow more concise answers and you should hopefully see how to go about generalizing from there. Cheers. $\endgroup$ – cardinal Mar 29 '12 at 23:39
  • $\begingroup$ No, {$X_n$} are not necessarily i.i.d. (in fact, isn't $T_X$ = $T_Y$ if they were?). I can see your point, but unfortunately, I am just trying to figure out the very general concept - so I can apply it in the future. Whoever answers this is welcome to narrow down to a reasonable case, but I'm really interested in as general an answer as possible.. $\endgroup$ – max Mar 30 '12 at 1:38
  • $\begingroup$ Thanks, max. Note that I asked if the $\{X_n\}$ were independent not if they were iid. So, can you clarify this point? In any case, the answer to the question in your comment is, in general, "no, they are not equal". Cheers. :) $\endgroup$ – cardinal Mar 30 '12 at 1:42
  • $\begingroup$ Ahh sorry! Yes they are independent! I updated the question. $\endgroup$ – max Mar 30 '12 at 2:20
  • 1
    $\begingroup$ Are 2 downvotes due to the fact that I cross-posted? I did read meta stackexchange on this topic, and it suggested it's ok if I link the questions and wait for a reasonable time before cross-posting. Given that the question was downvoted on the other forum too, I'll delete both of them. I just joined stats.stackexchange, and feel really unwelcome here. $\endgroup$ – max Apr 2 '12 at 15:54
1
$\begingroup$

Assume that each $X_n$ is distributed with mean $\mu_n$ and variance $\sigma^2_n$, for $n=1,..., N$.

For each $n$, we draw a sample of size $K_n$, denoted by $\hat{X}_{n,k}$, $k=1,...,K_n$. Let $M=\sum_{n=1}^N K_n$.

Then, for each sample we obtain the mean $$ \bar{\hat{X}}_{n} = \frac{1}{K_n} \sum_{k=1}^{K_n} \hat{X}_{n,k}. $$

Expected value and variance of each original mean

The expected value and variance of each $\bar{\hat{X}}_{n}$ is straightforward. We have $$E[\bar{\hat{X}}_n] = \mu_n$$ and $$\textrm{var}[\bar{\hat{X}}_n]= \frac{\sigma^2_n}{K_n}.$$

Expected value and variance of each bootstrap mean

Now consider the empirical distribution $\hat{F}$ obtained from the combined sample of $$\left\{\hat{X}_{1,1},...,\hat{X}_{1,K_1}, ... \hat{X}_{N,1},..., \hat{X}_{N,K_N}\right\}.$$ This is a mixture distribution. A draw from a random variable $Y_n \sim \hat{F}$ has expected value $$\mu = E[Y_n] = \sum_{\nu=1}^N \frac{K_\nu}{M} \mu_\nu$$ and variance $$ \textrm{var}[Y_n] = \sum_{\nu=1}^N \frac{K_\nu}{M} \left(\left( \mu_\nu - \mu \right)^2 + \sigma_\nu^2 \right). $$

Take a draw of size $K_n$ from $Y_n$. Note that the distribution of each $Y_n$ is the same. Then, as before, the sample mean is $$\bar{\hat{Y}}_n = \frac{1}{K_n} \sum_{k=1}^{K_n} \hat{Y}_{n,k}.$$

The expected value and variance of each $\bar{\hat{Y}}_n$ is $$E[\bar{\hat{Y}}_n] = \mu$$ and $$ \textrm{var}[\bar{\hat{Y}}_n] = \frac{1}{K_n} \sum_{\nu=1}^N \frac{K_\nu}{M} \left(\left( \mu_\nu - \mu \right)^2 + \sigma_\nu^2 \right). $$

Expected value and variance of the collection of original means

Consider that a draw from our collection $\left\{ \bar{\hat{X}}_1, ..., \bar{\hat{X}}_N \right\}$ is a draw from a population with a mixture distribution with equal weights $\frac1N$ and with expectation and variances for each component already determined. Therefore, the mean of this population is given by $$\bar\mu = \frac1N \sum_{n=1}^N \mu_n$$ and the variance of this population is given by $$ \frac1N \sum_{n=1}^N \left(\left(\mu_n - \bar\mu \right)^2 + \frac{\sigma^2_n}{K_n}\right)= \frac1N \sum_{n=1}^N \left(\mu_n - \bar\mu \right)^2 + \frac1N \sum_{n=1}^N \frac{\sigma^2_n}{K_n}.$$

Expected value and variance of the collection of bootstrap means

In a similar vein, we examine a draw from our collection $\left\{ \bar{\hat{Y}}_1, ..., \bar{\hat{Y}}_N \right\}$. Again, this is a mixture distribution with equal weights $1/N$ and with expectation and variances for each component already determined. Therefore, the mean of this population is $$\frac1N \sum_{n=1}^N \mu = \frac1N \sum_{n=1}^N \sum_{\nu=1}^N \frac{K_\nu}{M} \mu_\nu = \sum_{\nu=1}^N \frac{K_\nu}{M} \mu_\nu.$$ Note that this is the same as each individual expectation, so that the variance of this population is given by $$ \sum_{n=1}^N \frac1N \frac{1}{K_n} \sum_{\nu=1}^N \frac{K_\nu}{M} \left(\left( \mu_\nu - \mu \right)^2 + \sigma_\nu^2 \right), $$ which can be rewritten as $$ \frac1N \sum_{n=1}^N \frac{1}{K_n} \sum_{\nu=1}^N \frac{K_\nu}{M} \left( \mu_\nu - \mu \right)^2 + \frac1N \sum_{n=1}^N \frac{1}{K_n} \sum_{\nu=1}^N \frac{K_\nu}{M} \sigma_\nu^2. $$

Letting $$\frac{1}{H(K_1, ..., K_N)} = \frac1N \sum_{n=1}^N \frac{1}{K_n},$$ we see that $H=H(K_1,..., K_N)$ is the reciprocal of the arithmetic mean of the reciprocals, so that $H$ is the harmonic mean of the $K_n$. This allows us to rewrite the variance as $$ \frac{1}{H} \sum_{\nu=1}^N \frac{K_\nu}{M} \left( \mu_\nu - \mu \right)^2 + \frac{1}{H} \sum_{\nu=1}^N \frac{K_\nu}{M} \sigma_\nu^2. $$

Consequences

Since each $K_n \ge 1$, we have that $H \ge 1$, with equality only when all the $K_n=1$. Therefore, the reciprocal $1/H \le 1$ with equality only when all the $K_n=1$.

Comparing the variance of the original sample means with the variance of the bootstrap means, we see that they differ in two ways. First, the terms in the variance of the bootstrap means contain weights proportional to the $K_n$ rather than equal weighting. This includes the centering value for the term involving the expectations. Second, the terms in the variance of the bootstrap means are multiplied by the reciprocal of the harmonic mean of the $K_n$.

Suppose the $K_n=K\ge1$, $n=1,..., N$. That is, all the samples are of the same size. Then, $M=NK$, all the weights are equal to $1/N$, and the terms in summation for the bootstrap means are the same as those from the original samples. If $K=1$, then the variance of the bootstrap means is the same as the variance of the original sample means. If $K>1$, though, the variance of the bootstrap means is less than the variance of the original sample means due to the multiplier of the reciprocal of the harmonic mean.

Empirical check

Assuming any of this is accurate, we can look at some specific examples. Here is some code that should compute the given equations for the variance.

# Calculate the variance of the original means.

var.orig <- function(mu, s2, K) {

  N <- length(K)

  mu.bar <- mean(mu)

  v <- (1/N) * sum((mu - mu.bar)^2) + (1/N) * sum(s2/K)
  return(v)
}

# Calculate the variance of the bootstrap means.

var.boot <- function(mu, s2, K) {

  N <- length(K)
  M <- sum(K)

  mu.w <- sum( (K/M) * mu)

  v <- (1/N) * sum(1/K) * (
     sum( (K/M) * (mu - mu.w)^2 ) + sum( (K/M) * s2) )
  return(v)
}

Then, we can try some specific values.

# Set up some combinations.

Test <- data.frame(
    mu1 = c(0, 0, 0,  0,  0,  0,  0,  0,  0),
    mu2 = c(1, 1, 1,  1,  1,  1,  1,  1,  1),
    mu3 = c(2, 2, 2, 20,  2,  2,  2, 10,  1),
    si1 = c(1, 1, 5,  1,  1,  5, 10,  1,  3),
    si2 = c(1, 1, 5,  1,  1,  5,  1,  1,  1),
    si3 = c(1, 1, 5,  1,  1, 10,  1,  1, 20),
    K1  = c(1, 2, 2,  2, 90,  5, 20, 20, 10),
    K2  = c(1, 2, 2,  2,  1,  5,  2,  2,  5),
    K3  = c(1, 2, 2, 20, 20, 10,  2, 20, 30)
)

# Run through them.

for (i in 1:dim(Test)[1]) {

    mu <- c(Test$mu1[i], Test$mu2[i], Test$mu3[i])
        s2 <- c(Test$si1[i], Test$si2[i], Test$si3[i])
    K  <- c(Test$K1[i], Test$K2[i], Test$K3[i])

    Test$var.orig[i] <- var.orig(mu, s2, K)
        Test$var.boot[i] <- var.boot(mu, s2, K)
}

This yields the following results:

Test

#  mu1 mu2 mu3 si1 si2 si3 K1 K2 K3   var.orig   var.boot
#1   0   1   2   1   1   1  1  1  1  1.6666667  1.6666667
#2   0   1   2   1   1   1  2  2  2  1.1666667  0.8333333
#3   0   1   2   5   5   5  2  2  2  3.1666667  2.8333333
#4   0   1  20   1   1   1  2  2 20 85.0166667 18.8489583
#5   0   1   2   1   1   1 90  1 20  1.0203704  0.5635547
#6   0   1   2   5   5  10  5  5 10  1.6666667  1.3645833
#7   0   1   2  10   1   1 20  2  2  1.1666667  3.0989583
#8   0   1  10   1   1   1 20  2 20 20.4222222  5.1070295
#9   0   1   1   3   1  20 10  5 30  0.6111111  1.5871056

Empirically, picking larger samples from populations with higher variance tends to increase the bootstrap variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.