Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [resampling]

Resampling is taking a sample from a sample. Common uses are jackknifing (taking a subsample, eg all values but 1) & bootstrapping (sampling w/ replacement). These techniques can provide a robust estimate of a sampling distribution when it would be difficult or impossible to derive analytically.

72
votes
2answers
38k views

Resampling / simulation methods: monte carlo, bootstrapping, jackknifing, cross-validation, randomization tests, and permutation tests

I am trying to understand difference between different resampling methods (Monte Carlo simulation, parametric bootstrapping, non-parametric bootstrapping, jackknifing, cross-validation, randomization ...
33
votes
5answers
4k views

Can you overfit by training machine learning algorithms using CV/Bootstrap?

This question may well be too open-ended to get a definitive answer, but hopefully not. Machine learning algorithms, such as SVM, GBM, Random Forest etc, generally have some free parameters that, ...
29
votes
2answers
4k views

How well does bootstrapping approximate the sampling distribution of an estimator?

Having recently studied bootstrap, I came up with a conceptual question that still puzzles me: You have a population, and you want to know a population attribute, i.e. $\theta=g(P)$, where I use $P$ ...
25
votes
2answers
14k views

Why use stratified cross validation? Why does this not damage variance related benefit?

I've been told that is beneficial to use stratified cross validation especially when response classes are unbalanced. If one purpose of cross-validation is to help account for the randomness of our ...
20
votes
2answers
7k views

what are the assumptions of permutation test?

It's often stated that permutation tests have no assumptions, however this is certainly not true. For example if my samples are somehow correlated, I can imagine that permuting their labels would not ...
19
votes
2answers
16k views

Caret re-sampling methods

I am using the library caret in R to test various modelling procedures. The trainControl object allows one to specify a re-...
15
votes
2answers
11k views

Test for IID sampling

How would you test or check that sampling is IID (Independent and Identically Distributed)? Note that I do not mean Gaussian and Identically Distributed, just IID. And idea that comes to my mind is ...
14
votes
2answers
8k views

What is the procedure for “bootstrap validation” (a.k.a. “resampling cross-validation”)?

"Bootstrap validation"/"resampling cross-validation" is new to me, but was discussed by the answer to this question. I gather it involves 2 types of data: the real data and simulated data, where a ...
14
votes
1answer
457 views

Is this method of resampling time-series known in the literature? Does it have a name?

I was recently looking for ways to resample time series, in ways that Approximately preserve the auto-correlation of long memory processes. Preserve the domain of the observations (for instance a ...
13
votes
3answers
4k views

Why is bootstrapping useful?

If all you are doing is re-sampling from the empirical distribution, why not just study the empirical distribution? For example instead of studying the variability by repeated sampling, why not just ...
13
votes
2answers
2k views

Best suggested textbooks on Bootstrap resampling?

I just wanted to ask which are in your opinion the best available books on bootstrap out there. By this I don't necessarily only mean the one written by its developers. Could you please indicate ...
13
votes
2answers
9k views

Testing Classification on Oversampled Imbalance Data

I am working on severely imbalanced data. In literature, several methods are used to re-balance the data using re-sampling (over- or under-sampling). Two good approaches are: SMOTE: Synthetic ...
13
votes
1answer
1k views

Is centering needed when bootstrapping the sample mean?

When reading about how to approximate the distribution of the sample mean I came across the nonparametric bootstrap method. Apparently one can approximate the distribution of $\bar{X}_n-\mu$ by the ...
11
votes
3answers
12k views

How to resample in R without repeating permutations?

In R, if I set.seed(), and then use the sample function to randomize a list, can I guarantee I won't generate the same permutation? ie... ...
11
votes
1answer
3k views

Why not always use bootstrap CIs?

I was wondering how bootstrap CIs (and BCa in barticular) perform on normally-distributed data. There seems to be lots of work examining their performance on various types of distributions, but could ...
10
votes
1answer
2k views

Is bootstrapping appropriate for this continuous data?

I'm a complete newbie :) I'm doing a study with a sample size of 10,000 from a population of about 745,000. Each sample represents a "percentage similarity". The great majority of the samples are ...
10
votes
2answers
727 views

Good text for resampling?

Can the group recommend a good introduction text/resource to applied resampling techniques? Specifically, I am interested in alternatives to classical parametric tests (e.g. t tests, ANOVA, ANCOVA) ...
10
votes
4answers
411 views

Why do hypothesis tests on resampled datasets reject the null too often?

tl;dr: Starting with a dataset generated under the null, I resampled cases with replacement and conducted a hypothesis test on each resampled dataset. These hypothesis tests reject the null more than ...
10
votes
1answer
3k views

Gini coefficient and error bounds

I have a time series of data with N=14 counts at each time point, and I want to calculate the Gini coefficient and a standard error for this estimate at each time point. Since I have only N=14 counts ...
9
votes
2answers
5k views

Size of bootstrap samples

I'm learning about bootstrapping as a means of estimating the variance of a sample statistic. I have one basic doubt. Quoting from http://web.stanford.edu/class/psych252/tutorials/doBootstrapPrimer....
9
votes
1answer
2k views

Can bootstrap resampling be used to calculate a confidence interval for the variance of a dataset?

I know that if you re-sample from a data set many times and calculate the mean each time, these means will follow a normal distribution (by the CLT). Thus, you can calculate a confidence interval on ...
9
votes
2answers
2k views

Should I bootstrap at the cluster level or the individual level?

I have a survival model with patients nested in hospitals that includes a random-effect for the hospitals. The random effect is gamma-distributed, and I am trying to report the 'relevance' of this ...
8
votes
1answer
5k views

Bootstrap methodology. Why resample “with replacement” instead of random subsampling?

The bootstrap method has seen a great diffusion in the last years, I also use it a lot, especially because the reasoning behind is quite intuitive. But that's one thing I don't understand. Why Efron ...
8
votes
1answer
7k views

Oversampling with categorical variables

I would like to perform a combination of oversampling and undersampling in order to balance my dataset with roughly 4000 customers divided into two groups, where one of the groups have a proportion of ...
8
votes
1answer
183 views

What method is simulating pvalues from re sampling from the data

A while back I asked a question about correlating times between time stamps and received a response from Peter Ellis that said I could calculate mean distances between codes... This already will ...
8
votes
1answer
1k views

Control Function Approach and Bootstrap

Let's start assuming that I have cross-sectional data on $y$, $x_1$, $x_2$ (see below for $y$, $x_1$, $x_2$). I want to estimate the effect of variables $x_1$ and $x_2$ and their interaction ($x_3= ...
8
votes
2answers
2k views

Subsample of a random sample: random sample?

Let's say you have a large random sample of soccer players in Europe but you are only interested in what happens in Spain. Could you reduce your sample to players in Spain and still call it a random ...
7
votes
1answer
4k views

Required number of permutations for a permutation-based p-value

If I need to calculate a permutation-based $p$-value with significance level $\alpha$, how many permutations do I need ? From the article "Permutation Tests for Studying Classifier Performance", page ...
7
votes
2answers
11k views

By using SMOTE the classification of the validation set is bad

I want to do classification with 2 classes. When I classify without smote I get: ...
7
votes
3answers
4k views

Bootstrapping estimates of out-of-sample error

I know how to use bootstrap re-sampling to find confidence intervals for in-sample error or R2: ...
7
votes
1answer
1k views

Increase the sample size of a Latin Hypercube study?

I want to create a climate model ensemble, testing 5 parameters (real, uniformly distributed between two values), using a latin hypercube approach. The problem is that I'm not sure how many ...
7
votes
3answers
163 views

Enlarging a random sample

In our project we have a population of 1000+ individuals. We picked a random sample of 107 individuals, but then we realized we needed more precision, so now we want to have a larger sample. The ...
7
votes
1answer
374 views

Restricting minimum subgroup size in a bootstrap resampling study - why is this approach wrong?

I'm currently doing a simple re-sampling study where I compare different methods for generating the confidence interval for linear regression models. I'm trying to follow Burton et. al's (2006) ...
6
votes
2answers
443 views

What are good references on calculating confidence intervals using subsampling or the delete-d jackknife?

I searched for references on using subsampling or the delete-d jackknife to calculate confidence intervals but wasn't able to find much. Could someone please offer more reference on using subsampling ...
6
votes
2answers
197 views

Resampling within a survey to account for missing data

Suppose I have survey responses that look like this: ...
6
votes
1answer
473 views

How does pooling and resampling affect variance of sample mean?

Suppose I have $N$ independent random variables $X_n$. I draw a sample of predetermined size $K_n$ from each of them. Denote the average of each sample $\bar{\hat{X}}_n$, and the total number of ...
6
votes
0answers
741 views

Comparison of the jacknife vs the bootstrap

I am interested in understanding the relative pros and cons of bootstrap versus jacknife resampling. Both are used in iterative algorithmic approaches to estimating the precision of a prediction or ...
5
votes
1answer
169 views

Why are all the permutations of i.i.d. samples from a continuous distribution equally likely?

Suppose $X$ is i.i.d from a continuous distribution Why is$$P(X_{i_1}<X_{i_2}<\cdots<X_{i_3})=P(X_{j_1}<X_{j_2}<\cdots<X_{j_3})=\frac{1}{n!}$$for all $i,j$? I think we can reason ...
5
votes
2answers
197 views

Estimating error from a 1% sample

Say we decide to take a 1% random sample (without replacement) of a population to estimate how many individuals have some condition. We then observe that X individuals in the sample have this ...
5
votes
1answer
269 views

How to generate a more accurate distribution of sample of random variates?

I want to generate a sample of random variates choosing 5 out of 7 days of the week, $X_1,\ldots,X_5$, such that the aggregate counts resemble a given day-of-week profile, namely $$\mathbb{E}[\mathbb{...
5
votes
1answer
2k views

Shifting bootstrap confidence interval to be centered around original parameter

I've been doing a bit of research into bootstrapping as I've been told one method of performing it, and this seems to differ from what I can find in other sources. I have a sample, and want to ...
5
votes
1answer
514 views

Bias and variance estimation with boostrap

The Wikipedia article about Jacknife estimation of the bias and variance of an estimator $\theta$ includes the following formulas: Variance of $\theta$: $ \operatorname {Var}(\theta )=\sigma ^{2}={\...
5
votes
1answer
181 views

Bootstrapping the data to set up a prior

I am using a Gaussian model with a conjugate Normal-Inverse-Wishart (NIW) prior, as described here. The advantage of this approach is that the marginal likelihood $p(y)$, which is what I am interested ...
5
votes
1answer
1k views

Logistic Regression with Unbalanced Data [duplicate]

I have a binary dataset which is 99% in one class and 1% in the other class. I MUST create a logistic regression. I have read literature that says both using this dataset as is, or over/undersampling ...
5
votes
0answers
549 views

Block bootstrap for dependent data with unequal sampling intervals?

I have data from a natural archive (lake sediment). For various reasons it is usually impossible to sample the archive equally in time, and we end up with a time series where essentially we have ...
5
votes
1answer
823 views

Assessing the representativeness of population sampling

I am looking for some suggestions about assessing the representativeness of a particular dataset I am analyzing. In this dataset I am looking at the relationship between two variables (e.g., X and Y)...
4
votes
3answers
1k views

Practical difference between “sampling” and “re-sampling with replacement”

What is the difference between "sampling" and "re-sampling with replacement" from a practical point of view (I mean how do I code that) ? When one say: "sampling $\tilde{x}_t^{(i)}$ from $p(x_t | x_{...
4
votes
1answer
2k views

Bootstrapping in R using the boot {boot} and Boot {car}

I'm trying my hand at resampling techniques with a dataset I have, and I think either I'm missing a conceptual point with bootstrapping, or I'm doing something incorrectly in ...
4
votes
1answer
3k views

For bootstrapping, why does a higher subsample size lead to lower variance?

I've been working on a bootstrapping problem that's left me a little confused and wondering whether I'm doing things correctly. We have around 200 samples from a population of about 3,400, we want to ...
4
votes
1answer
2k views

Optimal sampling strategy for EFA, CFA and SEM

I'm wondering what should be the optimal sampling strategy for my dissertation research. I have four data sources (two open source software projects meta-repositories and two global startup databases)....