Questions tagged [resampling]

Resampling is taking a sample from a sample. Common uses are jackknifing (taking a subsample, eg all values but 1) & bootstrapping (sampling w/ replacement). These techniques can provide a robust estimate of a sampling distribution when it would be difficult or impossible to derive analytically.

Filter by
Sorted by
Tagged with
73
votes
2answers
39k views

Resampling / simulation methods: monte carlo, bootstrapping, jackknifing, cross-validation, randomization tests, and permutation tests

I am trying to understand difference between different resampling methods (Monte Carlo simulation, parametric bootstrapping, non-parametric bootstrapping, jackknifing, cross-validation, randomization ...
33
votes
5answers
4k views

Can you overfit by training machine learning algorithms using CV/Bootstrap?

This question may well be too open-ended to get a definitive answer, but hopefully not. Machine learning algorithms, such as SVM, GBM, Random Forest etc, generally have some free parameters that, ...
29
votes
2answers
4k views

How well does bootstrapping approximate the sampling distribution of an estimator?

Having recently studied bootstrap, I came up with a conceptual question that still puzzles me: You have a population, and you want to know a population attribute, i.e. $\theta=g(P)$, where I use $P$ ...
26
votes
2answers
15k views

Why use stratified cross validation? Why does this not damage variance related benefit?

I've been told that is beneficial to use stratified cross validation especially when response classes are unbalanced. If one purpose of cross-validation is to help account for the randomness of our ...
20
votes
2answers
7k views

what are the assumptions of permutation test?

It's often stated that permutation tests have no assumptions, however this is certainly not true. For example if my samples are somehow correlated, I can imagine that permuting their labels would not ...
19
votes
2answers
16k views

Caret re-sampling methods

I am using the library caret in R to test various modelling procedures. The trainControl object allows one to specify a re-...
15
votes
2answers
10k views

Testing Classification on Oversampled Imbalance Data

I am working on severely imbalanced data. In literature, several methods are used to re-balance the data using re-sampling (over- or under-sampling). Two good approaches are: SMOTE: Synthetic ...
15
votes
2answers
12k views

Test for IID sampling

How would you test or check that sampling is IID (Independent and Identically Distributed)? Note that I do not mean Gaussian and Identically Distributed, just IID. And idea that comes to my mind is ...
14
votes
2answers
8k views

What is the procedure for “bootstrap validation” (a.k.a. “resampling cross-validation”)?

"Bootstrap validation"/"resampling cross-validation" is new to me, but was discussed by the answer to this question. I gather it involves 2 types of data: the real data and simulated data, where a ...
14
votes
1answer
465 views

Is this method of resampling time-series known in the literature? Does it have a name?

I was recently looking for ways to resample time series, in ways that Approximately preserve the auto-correlation of long memory processes. Preserve the domain of the observations (for instance a ...
13
votes
3answers
4k views

Why is bootstrapping useful?

If all you are doing is re-sampling from the empirical distribution, why not just study the empirical distribution? For example instead of studying the variability by repeated sampling, why not just ...
13
votes
2answers
2k views

Best suggested textbooks on Bootstrap resampling?

I just wanted to ask which are in your opinion the best available books on bootstrap out there. By this I don't necessarily only mean the one written by its developers. Could you please indicate ...
13
votes
1answer
1k views

Is centering needed when bootstrapping the sample mean?

When reading about how to approximate the distribution of the sample mean I came across the nonparametric bootstrap method. Apparently one can approximate the distribution of $\bar{X}_n-\mu$ by the ...
12
votes
1answer
3k views

Why not always use bootstrap CIs?

I was wondering how bootstrap CIs (and BCa in barticular) perform on normally-distributed data. There seems to be lots of work examining their performance on various types of distributions, but could ...
11
votes
3answers
12k views

How to resample in R without repeating permutations?

In R, if I set.seed(), and then use the sample function to randomize a list, can I guarantee I won't generate the same permutation? ie... ...
10
votes
1answer
2k views

Is bootstrapping appropriate for this continuous data?

I'm a complete newbie :) I'm doing a study with a sample size of 10,000 from a population of about 745,000. Each sample represents a "percentage similarity". The great majority of the samples are ...
10
votes
2answers
742 views

Good text for resampling?

Can the group recommend a good introduction text/resource to applied resampling techniques? Specifically, I am interested in alternatives to classical parametric tests (e.g. t tests, ANOVA, ANCOVA) ...
10
votes
4answers
426 views

Why do hypothesis tests on resampled datasets reject the null too often?

tl;dr: Starting with a dataset generated under the null, I resampled cases with replacement and conducted a hypothesis test on each resampled dataset. These hypothesis tests reject the null more than ...
10
votes
2answers
2k views

Should I bootstrap at the cluster level or the individual level?

I have a survival model with patients nested in hospitals that includes a random-effect for the hospitals. The random effect is gamma-distributed, and I am trying to report the 'relevance' of this ...
10
votes
1answer
3k views

Gini coefficient and error bounds

I have a time series of data with N=14 counts at each time point, and I want to calculate the Gini coefficient and a standard error for this estimate at each time point. Since I have only N=14 counts ...
9
votes
2answers
5k views

Size of bootstrap samples

I'm learning about bootstrapping as a means of estimating the variance of a sample statistic. I have one basic doubt. Quoting from http://web.stanford.edu/class/psych252/tutorials/doBootstrapPrimer....
9
votes
1answer
2k views

Can bootstrap resampling be used to calculate a confidence interval for the variance of a dataset?

I know that if you re-sample from a data set many times and calculate the mean each time, these means will follow a normal distribution (by the CLT). Thus, you can calculate a confidence interval on ...
9
votes
1answer
6k views

Bootstrap methodology. Why resample “with replacement” instead of random subsampling?

The bootstrap method has seen a great diffusion in the last years, I also use it a lot, especially because the reasoning behind is quite intuitive. But that's one thing I don't understand. Why Efron ...
8
votes
1answer
4k views

Required number of permutations for a permutation-based p-value

If I need to calculate a permutation-based $p$-value with significance level $\alpha$, how many permutations do I need ? From the article "Permutation Tests for Studying Classifier Performance", page ...
8
votes
1answer
8k views

Oversampling with categorical variables

I would like to perform a combination of oversampling and undersampling in order to balance my dataset with roughly 4000 customers divided into two groups, where one of the groups have a proportion of ...
8
votes
1answer
185 views

What method is simulating pvalues from re sampling from the data

A while back I asked a question about correlating times between time stamps and received a response from Peter Ellis that said I could calculate mean distances between codes... This already will ...
8
votes
1answer
2k views

Control Function Approach and Bootstrap

Let's start assuming that I have cross-sectional data on $y$, $x_1$, $x_2$ (see below for $y$, $x_1$, $x_2$). I want to estimate the effect of variables $x_1$ and $x_2$ and their interaction ($x_3= ...
8
votes
2answers
2k views

Subsample of a random sample: random sample?

Let's say you have a large random sample of soccer players in Europe but you are only interested in what happens in Spain. Could you reduce your sample to players in Spain and still call it a random ...
7
votes
2answers
11k views

By using SMOTE the classification of the validation set is bad

I want to do classification with 2 classes. When I classify without smote I get: ...
7
votes
3answers
4k views

Bootstrapping estimates of out-of-sample error

I know how to use bootstrap re-sampling to find confidence intervals for in-sample error or R2: ...
7
votes
1answer
1k views

Increase the sample size of a Latin Hypercube study?

I want to create a climate model ensemble, testing 5 parameters (real, uniformly distributed between two values), using a latin hypercube approach. The problem is that I'm not sure how many ...
7
votes
3answers
165 views

Enlarging a random sample

In our project we have a population of 1000+ individuals. We picked a random sample of 107 individuals, but then we realized we needed more precision, so now we want to have a larger sample. The ...
7
votes
1answer
381 views

Restricting minimum subgroup size in a bootstrap resampling study - why is this approach wrong?

I'm currently doing a simple re-sampling study where I compare different methods for generating the confidence interval for linear regression models. I'm trying to follow Burton et. al's (2006) ...
7
votes
0answers
783 views

Comparison of the jacknife vs the bootstrap

I am interested in understanding the relative pros and cons of bootstrap versus jacknife resampling. Both are used in iterative algorithmic approaches to estimating the precision of a prediction or ...
6
votes
2answers
448 views

What are good references on calculating confidence intervals using subsampling or the delete-d jackknife?

I searched for references on using subsampling or the delete-d jackknife to calculate confidence intervals but wasn't able to find much. Could someone please offer more reference on using subsampling ...
6
votes
2answers
198 views

Resampling within a survey to account for missing data

Suppose I have survey responses that look like this: ...
6
votes
1answer
186 views

Bootstrapping the data to set up a prior

I am using a Gaussian model with a conjugate Normal-Inverse-Wishart (NIW) prior, as described here. The advantage of this approach is that the marginal likelihood $p(y)$, which is what I am interested ...
6
votes
1answer
484 views

How does pooling and resampling affect variance of sample mean?

Suppose I have $N$ independent random variables $X_n$. I draw a sample of predetermined size $K_n$ from each of them. Denote the average of each sample $\bar{\hat{X}}_n$, and the total number of ...
5
votes
1answer
180 views

Why are all the permutations of i.i.d. samples from a continuous distribution equally likely?

Suppose $X$ is i.i.d from a continuous distribution Why is$$P(X_{i_1}<X_{i_2}<\cdots<X_{i_3})=P(X_{j_1}<X_{j_2}<\cdots<X_{j_3})=\frac{1}{n!}$$for all $i,j$? I think we can reason ...
5
votes
2answers
197 views

Estimating error from a 1% sample

Say we decide to take a 1% random sample (without replacement) of a population to estimate how many individuals have some condition. We then observe that X individuals in the sample have this ...
5
votes
1answer
270 views

How to generate a more accurate distribution of sample of random variates?

I want to generate a sample of random variates choosing 5 out of 7 days of the week, $X_1,\ldots,X_5$, such that the aggregate counts resemble a given day-of-week profile, namely $$\mathbb{E}[\mathbb{...
5
votes
1answer
2k views

Shifting bootstrap confidence interval to be centered around original parameter

I've been doing a bit of research into bootstrapping as I've been told one method of performing it, and this seems to differ from what I can find in other sources. I have a sample, and want to ...
5
votes
1answer
538 views

Bias and variance estimation with boostrap

The Wikipedia article about Jacknife estimation of the bias and variance of an estimator $\theta$ includes the following formulas: Variance of $\theta$: $ \operatorname {Var}(\theta )=\sigma ^{2}={\...
5
votes
1answer
825 views

Oversampling: whole set or training set

I have a rather small dataset of 4 000 points (140 features) to feed to a NN binary classifier. The problem is only ~700 of them represent the second class. Is it more common to resample the whole ...
5
votes
1answer
1k views

Logistic Regression with Unbalanced Data [duplicate]

I have a binary dataset which is 99% in one class and 1% in the other class. I MUST create a logistic regression. I have read literature that says both using this dataset as is, or over/undersampling ...
5
votes
0answers
566 views

Block bootstrap for dependent data with unequal sampling intervals?

I have data from a natural archive (lake sediment). For various reasons it is usually impossible to sample the archive equally in time, and we end up with a time series where essentially we have ...
5
votes
1answer
832 views

Assessing the representativeness of population sampling

I am looking for some suggestions about assessing the representativeness of a particular dataset I am analyzing. In this dataset I am looking at the relationship between two variables (e.g., X and Y)...
4
votes
1answer
4k views

sampling/importance resampling - why resample?

I'm trying to understand the SIR algorithm. In order to do so it seemed best to look at some existing r code (see below), taken from http://hedibert.org/wp-content/uploads/2013/12/example-iii.R.txt. <...
4
votes
3answers
1k views

Practical difference between “sampling” and “re-sampling with replacement”

What is the difference between "sampling" and "re-sampling with replacement" from a practical point of view (I mean how do I code that) ? When one say: "sampling $\tilde{x}_t^{(i)}$ from $p(x_t | x_{...
4
votes
1answer
2k views

Bootstrapping in R using the boot {boot} and Boot {car}

I'm trying my hand at resampling techniques with a dataset I have, and I think either I'm missing a conceptual point with bootstrapping, or I'm doing something incorrectly in ...