I need to validate a specific/trained neural network for classification, and I'm planning to use bootstrapping for this purpose. My idea is to keep fixed the trained network and generate bootstrap samples only for the test set, and then obtain statistics about accuracy/false pos/false neg for that fixed network.

Do you think this approach is adequate? I'm asking because, as far as I've seen, the typical approach is to train the network at each bootstrap sample (after holding out the test samples). Instead, I want to obtain statistics about classification accuracy for a specific network.

Another question: in my case, the samples are obtained uniformly from $\mathbb{R}^n$, unlike the typical case when you resample from a finite set of observations. Then, is it still correct to speak of "bootstrapping"?

Thank you!

up vote 0 down vote accepted

Bootstrap is a statistical tool help us emulate the process of acquiring new sample set, usually when it is impossible or inconvenient to do so. It resamples from an existing sample set: If you are resampling from the training set to generate the testing sets, it will not be adequate since all these testing cases have been used during your training; if you are resampling from $\mathbb{R}^n$, this is randomly, independently generating test set from the population, which is adequate, but cannot be called bootstrap.

  • Many thanks for clarifying the meaning of bootstrap. What do you think of the validation scheme I described (i.e., resample only the test set from $\mathbb{R}^n$ and keep the network fixed)? – Nicola Paoletti Oct 12 '17 at 20:44
  • It looks good. But a more standard way is to obtain data needed for your model(training/testing) in the first place. For example obtain 10,000 samples. Then divided them into training and testing data. (7,000 for training and 3,000 for testing). From my understanding, what you are doing is training the model used the obtained 7,000 samples and then sampling another 3,000 data from $\mathbb{R}^n$ for testing. This is a little bit odd, but will not cause any problem for validating your model. – X. Zhang Oct 12 '17 at 20:56
  • Yes, just to confirm, the workflow would be: 1. Train the model using samples drawn from $\mathbb{R}^n$; 2. Sample N times (e.g. N=1000) the testing set from $\mathbb{R}^n$; 3. Obtain statistics (e.g. mean accuracy and S.E.) by validating the trained model with all test datasets – Nicola Paoletti Oct 12 '17 at 21:57
  • This answer does not really say much about the bootstrap. It involves resampling from the original data set with replacement. It is used when parametric assumptions are questionable and it often is used because it has better properties then alternative methods. – Michael Chernick Oct 12 '17 at 22:36

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.