Linked Questions

42
votes
5answers
15k views

Why on average does each bootstrap sample contain roughly two thirds of observations?

I have run across the assertion that each bootstrap sample (or bagged tree) will contain on average approximately $2/3$ of the observations. I understand that the chance of not being selected in any ...
83
votes
3answers
12k views

What are examples where a “naive bootstrap” fails?

Suppose I have a set of sample data from an unknown or complex distribution, and I want to perform some inference on a statistic $T$ of the data. My default inclination is to just generate a bunch of ...
48
votes
1answer
27k views

Bootstrap vs. jackknife

Both bootstrap and jackknife methods can be used to estimate bias and standard error of an estimate and mechanisms of both resampling methods are not huge different: sampling with replacement vs. ...
2
votes
3answers
1k views

Implementing the 0.632+ bootstrap method using the Weka Java API

I am trying to implement the 0.632+ bootstrap estimator (as proposed by Efron and Tibshirani 1997) in order to perform certain benchmarks and compare it with other cross-validation methods, such as ...
1
vote
1answer
999 views

Differences between cross validation and bootstrapping to estimate the standard error of the AUC of a given ROC curve

I know there's been some discussion on differences between CV and bootstrapping for estimating out-of-sample prediction error of a classifier. For example, in here (Differences between cross ...
2
votes
1answer
489 views

Are studentized deleted residuals a form of k-fold cross validation when K=N?

Are Studentized deleted residuals a form of k-fold cross validation when K=N? (this question is asked in the context of the discussion here)
1
vote
2answers
653 views

LIBSVM overfitting

I trained two svms (LIBSVM) with 15451 samples after I did a 10-fold cross-validation and found the best parameter values for gamma and C (RBF kernel). In one svm I used just 1 feature and in the ...
11
votes
1answer
250 views

What if probabilities are not equal in the “.632 Rule?”

This question is derived from this one about the ".632 Rule." I am writing with particular reference to user603's answer/notation to the extent it simplifies matters. That answer begins with a ...
1
vote
2answers
410 views

Evaluate the performance of a model with bootstrap

This question is about the application of the bootstrap rule The population is to the sample as the sample is to the bootstrap samples.I have a small dataset about lung cancer.There are 160 patients ...
3
votes
1answer
312 views

What is “Adjusted CV” or “Bias-corrected CV”?

In the documentation for the R package pls, the following statement appears in the help file for the MSEP function: "CV" is ...
1
vote
1answer
179 views

Approaching the limit in the .632 rule when n is unknown

This question relates to the ".632 rule" asked about in an earlier question here. My question is, assuming we have a process that approximates the random-sample-with-replacement for large (unknown) n ...
2
votes
0answers
120 views

Optimism bootstrap with non-linear models

I have come across an example in my research with heavily overfit non-linear probabilistic classifiers, where the optimism bootstrap appears to underestimate the optimism, even when using a proper ...
2
votes
1answer
45 views

What value of alpha should I choose regularization

What value of alpha should I choose in glmnet? Should I use one which minimizes the cross-validation error, one which is one standard deviation above or below the one which gives the best error (like ...
0
votes
0answers
34 views

Sample size of each resample

I am calculating a performance metric based on the resamples obtained from cross-validation and bootstrap using the R-package caret: ...