56 votes
Accepted

Bootstrapping vs Bayesian Bootstrapping conceptually?

The (frequentist) bootstrap takes the data as a reasonable approximation to the unknown population distribution. Therefore, the sampling distribution of a statistic (a function of the data) can be ...
user avatar
  • 3,011
48 votes
Accepted

Why is the sampling distribution of variance a chi-squared distribution?

[I'll assume from the discussion in your question that you're happy to accept as fact that if $Z_i, i=1,2,\ldots,k$ are independent identically distributed $N(0,1)$ random variables then $\sum_{i=1}^{...
user avatar
  • 263k
46 votes
Accepted

What is the name of the statistical fallacy whereby outcomes of previous coin flips influence beliefs about subsequent coin flips?

It's called the Gambler's fallacy.
user avatar
  • 2,000
38 votes
Accepted

What is the standard error of the sample standard deviation?

Let $\mu_4 = E(X-\mu)^4$. Then, the formula for the SE of $s^2$ is: $$ se(s^2) = \sqrt{ \frac{1}{n}\left(\mu_4 -\frac{n-3}{n-1} \sigma^4\right)} $$ This is an exact formula, valid for any sample ...
user avatar
36 votes

Why does increasing the sample size lower the (sampling) variance?

Standard deviations of averages are smaller than standard deviations of individual observations. [Here I will assume independent identically distributed observations with finite population variance; ...
user avatar
  • 263k
35 votes

What is the name of the statistical fallacy whereby outcomes of previous coin flips influence beliefs about subsequent coin flips?

The first sentence of this question, incorporates another (related) fallacy: "As we all know, if you flip a coin that has an equal chance of landing heads as it does tails, then if you flip the ...
user avatar
32 votes

What percentage of a population needs a test in order to estimate prevalence of a disease? Say, COVID-19

1) Making some assumptions about the population size (namely that it is large enough that a binomial model is appropriate), the prevalence of a disease in a population at a particular time can be ...
user avatar
31 votes
Accepted

Computation of the marginal likelihood from MCMC samples

The extension by Chib and Jeliazkov (2001) unfortunately gets quickly costly or highly variable, which is a reason why it is not much used outside Gibbs sampling cases. While there are many ways and ...
user avatar
  • 93.6k
30 votes
Accepted

The paradox of i.i.d. data (at least for me)

I think you are confusing an estimated model of a distribution with a random variable. Let's rewrite the independence assumption as follows: $$ P(X_n | \theta, X_{i_1}, X_{i_2}, \dots, X_{i_k}) = P(...
user avatar
  • 2,161
30 votes

What if a non-random sample is identical to a random sample?

Play poker with your friend, bet a lot of money, and cheat to give yourself a royal flush (it beats every other hand). “That’s cheating!” “Nah, it’s one of the possible hands. Pay up.” Yes, it’s about ...
user avatar
  • 33.9k
30 votes

The "Amazing Hidden Power" of Random Search?

One limitation of random search is that searching over a large space is extremely challenging; even a small difference can spoil the result. Émile Borel's 1913 article "Mécanique Statistique et ...
user avatar
  • 80.4k
27 votes
Accepted

Generating data with a given sample covariance matrix

There are two different typical situations for these kind of problems: i) you want to generate a sample from a given distribution whose population characteristics match the ones specified (but due to ...
user avatar
  • 263k
26 votes
Accepted

Is low bias in a sample a synonym for high variance?

No. You can have both high or both low at same time. Here is an illustrate example. picture and article source I also recommend you to read the article where this picture comes from. The reason you ...
user avatar
  • 33.8k
25 votes

Generating data with a given sample covariance matrix

@Glen_b gave a good answer (+1), which I want to illustrate with some code. How to generate $n$ samples from a $d$-dimensional multivariate Gaussian distribution with a given covariance matrix $\...
user avatar
  • 95.8k
24 votes
Accepted

Size of bootstrap samples

Bootstrap is conducted by sampling with replacement. It seems that the term "with replacement" is unclear for you. As noted by whuber, illustration of sampling with replacement is given on p. 3 of the ...
user avatar
  • 117k
23 votes
Accepted

Can I use moments of a distribution to sample the distribution?

Three moments don't determine a distributional form; if you choose a distribution-famiy with three parameters which relate to the first three population moments, you can do moment matching ("method of ...
user avatar
  • 263k
23 votes
Accepted

Why we don’t make use of the t-distribution for constructing a confidence interval for a proportion?

Both the standard Normal and Student t distributions are rather poor approximations to the distribution of $$Z = \frac{\hat p - p}{\sqrt{\hat p(1-\hat p)/n}}$$ for small $n,$ so poor that the error ...
user avatar
  • 294k
22 votes
Accepted

Best suggested textbooks on Bootstrap resampling?

There are two "classic" ones: Efron, B. & Tibshirani, R. J. (1993). An introduction to the bootstrap. London: Chapman & Hall/CRC. Davison, A. C. & Hinkley, D. V. (2009). Bootstrap ...
22 votes
Accepted

Is the mean of samples still a valid sample?

No, $\bar x$ has its own sampling distribution. Take, for example, the variances of $\bar x$ and $x_i$, in which the former is always lower ($\leq$) than the latter, which means $\bar x$ is not ...
user avatar
  • 52.6k
21 votes

Is the mean of samples still a valid sample?

Good examples so far but consider $$X_i \sim Bernoulli(.5)$$ In that case the distribution of the data will only have support on 0 and 1. But the sample mean will have an ever decreasing probability ...
user avatar
  • 2,112
21 votes

If a sample is normally distributed, is its population always normally distributed?

I'll be talking somewhat loosely here, but hopefully this can give a way of thinking about it. Once you observe it, a sample is a collection of numbers, not a collection of random variables (with ...
user avatar
  • 263k
19 votes
Accepted

Why is bootstrapping useful?

Bootstrapping (or other resampling) is an experimental method to estimate the distribution of a statistic. It is a very straightforward and easy method (it just means you compute with many random ...
user avatar
19 votes

The "Amazing Hidden Power" of Random Search?

Consider a neural network model with 100 weights. If we think only about getting the sign of the weights right and don't worry for the moment about their magnitude. There are 2^100 combinations of ...
user avatar
19 votes

How can I reduce a very large sample size for statistical significance (sampling methods)?

Hypothesis testing with p-values is useful in some situations where you need to make a crisp decision from one experiment. You don't. Instead of worrying about p-values, compute confidence intervals ...
user avatar
18 votes

How to choose the training, cross-validation, and test set sizes for small sample-size data?

You surely found the very similar question: Choice of K in K-fold cross-validation ? (Including the link to Ron Kohavi's work) If your sample size is already small I recommend avoiding any data driven ...
user avatar
18 votes
Accepted

How to generate a $\pm 1$ sequence with mean $0.05$?

Your desired mean is given by equation: $\frac{N\cdot p - N \cdot (1-p)}{N} = .05$ from which follows that the probability of the 1s should be ...
user avatar
18 votes

Why do several (if not all) parametric hypothesis tests assume random sampling?

If you are not making any inference for a wider group than your actual sample, then there is no application of statistical tests in the first place, and the question of "bias" does not arise. In this ...
user avatar
  • 100k
17 votes
Accepted

Random Forests out-of-bag sample size

It comes from the construction of a bootstrap sample: you're sampling $n$ observations with replacement to a sample size of $n$. The probability that an observation is omitted is $(1-\frac{1}{n})^n.$* ...
user avatar
  • 80.4k
17 votes
Accepted

Chance that bootstrap sample is exactly the same as the original sample

Note that at each observation position ($i=1, 2, ..., n$) we can choose any of the $n$ observations, so there are $n^n$ possible resamples (keeping the order in which they are drawn) of which $n!$ are ...
user avatar
  • 263k

Only top scored, non community-wiki answers of a minimum length are eligible