Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [resampling]

Resampling is taking a sample from a sample. Common uses are jackknifing (taking a subsample, eg all values but 1) & bootstrapping (sampling w/ replacement). These techniques can provide a robust estimate of a sampling distribution when it would be difficult or impossible to derive analytically.

72
votes
2answers
38k views

Resampling / simulation methods: monte carlo, bootstrapping, jackknifing, cross-validation, randomization tests, and permutation tests

I am trying to understand difference between different resampling methods (Monte Carlo simulation, parametric bootstrapping, non-parametric bootstrapping, jackknifing, cross-validation, randomization ...
13
votes
2answers
2k views

Best suggested textbooks on Bootstrap resampling?

I just wanted to ask which are in your opinion the best available books on bootstrap out there. By this I don't necessarily only mean the one written by its developers. Could you please indicate ...
33
votes
5answers
4k views

Can you overfit by training machine learning algorithms using CV/Bootstrap?

This question may well be too open-ended to get a definitive answer, but hopefully not. Machine learning algorithms, such as SVM, GBM, Random Forest etc, generally have some free parameters that, ...
13
votes
2answers
9k views

Testing Classification on Oversampled Imbalance Data

I am working on severely imbalanced data. In literature, several methods are used to re-balance the data using re-sampling (over- or under-sampling). Two good approaches are: SMOTE: Synthetic ...
25
votes
2answers
14k views

Why use stratified cross validation? Why does this not damage variance related benefit?

I've been told that is beneficial to use stratified cross validation especially when response classes are unbalanced. If one purpose of cross-validation is to help account for the randomness of our ...
19
votes
2answers
16k views

Caret re-sampling methods

I am using the library caret in R to test various modelling procedures. The trainControl object allows one to specify a re-...
9
votes
2answers
5k views

Size of bootstrap samples

I'm learning about bootstrapping as a means of estimating the variance of a sample statistic. I have one basic doubt. Quoting from http://web.stanford.edu/class/psych252/tutorials/doBootstrapPrimer....
8
votes
2answers
2k views

Subsample of a random sample: random sample?

Let's say you have a large random sample of soccer players in Europe but you are only interested in what happens in Spain. Could you reduce your sample to players in Spain and still call it a random ...
14
votes
2answers
8k views

What is the procedure for “bootstrap validation” (a.k.a. “resampling cross-validation”)?

"Bootstrap validation"/"resampling cross-validation" is new to me, but was discussed by the answer to this question. I gather it involves 2 types of data: the real data and simulated data, where a ...
20
votes
2answers
7k views

what are the assumptions of permutation test?

It's often stated that permutation tests have no assumptions, however this is certainly not true. For example if my samples are somehow correlated, I can imagine that permuting their labels would not ...
8
votes
1answer
5k views

Bootstrap methodology. Why resample “with replacement” instead of random subsampling?

The bootstrap method has seen a great diffusion in the last years, I also use it a lot, especially because the reasoning behind is quite intuitive. But that's one thing I don't understand. Why Efron ...
7
votes
1answer
4k views

Required number of permutations for a permutation-based p-value

If I need to calculate a permutation-based $p$-value with significance level $\alpha$, how many permutations do I need ? From the article "Permutation Tests for Studying Classifier Performance", page ...
7
votes
3answers
4k views

Bootstrapping estimates of out-of-sample error

I know how to use bootstrap re-sampling to find confidence intervals for in-sample error or R2: ...
5
votes
1answer
169 views

Why are all the permutations of i.i.d. samples from a continuous distribution equally likely?

Suppose $X$ is i.i.d from a continuous distribution Why is$$P(X_{i_1}<X_{i_2}<\cdots<X_{i_3})=P(X_{j_1}<X_{j_2}<\cdots<X_{j_3})=\frac{1}{n!}$$for all $i,j$? I think we can reason ...
10
votes
2answers
727 views

Good text for resampling?

Can the group recommend a good introduction text/resource to applied resampling techniques? Specifically, I am interested in alternatives to classical parametric tests (e.g. t tests, ANOVA, ANCOVA) ...
9
votes
2answers
2k views

Should I bootstrap at the cluster level or the individual level?

I have a survival model with patients nested in hospitals that includes a random-effect for the hospitals. The random effect is gamma-distributed, and I am trying to report the 'relevance' of this ...
1
vote
1answer
139 views

how to apply bootstrapping in this case?

I have a small sample of marks obtained by a group of students and I would like to apply a binomial test to check if more than half of the size of the classroom has failed due to the application of a ...
29
votes
2answers
4k views

How well does bootstrapping approximate the sampling distribution of an estimator?

Having recently studied bootstrap, I came up with a conceptual question that still puzzles me: You have a population, and you want to know a population attribute, i.e. $\theta=g(P)$, where I use $P$ ...
15
votes
2answers
11k views

Test for IID sampling

How would you test or check that sampling is IID (Independent and Identically Distributed)? Note that I do not mean Gaussian and Identically Distributed, just IID. And idea that comes to my mind is ...
10
votes
1answer
2k views

Is bootstrapping appropriate for this continuous data?

I'm a complete newbie :) I'm doing a study with a sample size of 10,000 from a population of about 745,000. Each sample represents a "percentage similarity". The great majority of the samples are ...
7
votes
2answers
11k views

By using SMOTE the classification of the validation set is bad

I want to do classification with 2 classes. When I classify without smote I get: ...
2
votes
1answer
2k views

Compute confidence interval for univariate Kernel Density Estimator

I've got a univariate dataset (timeseries) for two kind of simulated systems, and I want to explore the differences between the two. To do that, I can build a univariate gaussian KDE for each dataset ...
11
votes
3answers
12k views

How to resample in R without repeating permutations?

In R, if I set.seed(), and then use the sample function to randomize a list, can I guarantee I won't generate the same permutation? ie... ...
13
votes
1answer
1k views

Is centering needed when bootstrapping the sample mean?

When reading about how to approximate the distribution of the sample mean I came across the nonparametric bootstrap method. Apparently one can approximate the distribution of $\bar{X}_n-\mu$ by the ...
10
votes
1answer
3k views

Gini coefficient and error bounds

I have a time series of data with N=14 counts at each time point, and I want to calculate the Gini coefficient and a standard error for this estimate at each time point. Since I have only N=14 counts ...
10
votes
4answers
411 views

Why do hypothesis tests on resampled datasets reject the null too often?

tl;dr: Starting with a dataset generated under the null, I resampled cases with replacement and conducted a hypothesis test on each resampled dataset. These hypothesis tests reject the null more than ...
9
votes
1answer
2k views

Can bootstrap resampling be used to calculate a confidence interval for the variance of a dataset?

I know that if you re-sample from a data set many times and calculate the mean each time, these means will follow a normal distribution (by the CLT). Thus, you can calculate a confidence interval on ...
7
votes
1answer
1k views

Increase the sample size of a Latin Hypercube study?

I want to create a climate model ensemble, testing 5 parameters (real, uniformly distributed between two values), using a latin hypercube approach. The problem is that I'm not sure how many ...
3
votes
1answer
811 views

Addressing Non-response in a Convenience Sample

I am studying customer satisfaction in a large hierarchical organization. I plan to administer a voluntary survey to customers across the organization, and need to address non-response in my analysis. ...
2
votes
1answer
226 views

Why resample by centering vs. the sample estimate, not null parameter value?

In a classic paper on conducting bootstrapped hypothesis tests, Hall & Wilson (1991) present the following guideline, which I am trying to understand: On first read, this seemed obvious: $\...
5
votes
1answer
823 views

Assessing the representativeness of population sampling

I am looking for some suggestions about assessing the representativeness of a particular dataset I am analyzing. In this dataset I am looking at the relationship between two variables (e.g., X and Y)...
4
votes
1answer
2k views

Optimal sampling strategy for EFA, CFA and SEM

I'm wondering what should be the optimal sampling strategy for my dissertation research. I have four data sources (two open source software projects meta-repositories and two global startup databases)....
3
votes
2answers
1k views

R randomForest R replace=TRUE pro's and con's

When using R randomForest package I use replace=TRUE, which then dictates to: if (replace) nrow(x) else ceiling(.632*nrow(x)) I was wondering if anyone knows of ...
1
vote
1answer
783 views

How to resample matrices to test for the robustness of their correlation?

I have several populations where I have morphology and diet for each individual. I am interested in the correlation between diet and morphological distances. However the number of individuals in each ...
6
votes
2answers
443 views

What are good references on calculating confidence intervals using subsampling or the delete-d jackknife?

I searched for references on using subsampling or the delete-d jackknife to calculate confidence intervals but wasn't able to find much. Could someone please offer more reference on using subsampling ...
1
vote
1answer
213 views

Resampling under the null versus the alternative hypothesis

I'm looking at a community of organisms using simultaneous GLMs via the mvabund package. The manyglm function from this package ...
1
vote
1answer
1k views

Resampling correlated data using bootstrap

I have a dataset of $n×m$ numbers, where the $m$ variables ($m$ is in the order of 5–10) exhibit various degrees of correlation with one another. The format of the data is not necessarily a timeseries,...
1
vote
1answer
103 views

Random sample of a random sample from a population: Also a random sample?

From population P we draw an adequately sized random sample S1. From the sample S1 we draw an adequately sized random sample S2 (with replacement). Are the distributional characteristics of S2 ...
0
votes
0answers
287 views

Is the p value in a permutation test the same as the p value in the original test?

This is an extension from my last post. It seems to be too long to discuss there. When testing some null $H$ versus alternative $K$ by a test statistic $U(X)$, the p-value for $U$ on a sample $X$ ...
-6
votes
1answer
575 views

Did Normal distribution theory originate from psychometric analysis especially reliability and validity analysis? [closed]

The basic statistical literature does not talk much about the exact background and history of the normal distribution theory which provides us wth a base for traditional(descriptive analysis or ...