2
$\begingroup$

I need to validate a specific/trained neural network for classification, and I'm planning to use bootstrapping for this purpose. My idea is to keep fixed the trained network and generate bootstrap samples only for the test set, and then obtain statistics about accuracy/false pos/false neg for that fixed network.

Do you think this approach is adequate? I'm asking because, as far as I've seen, the typical approach is to train the network at each bootstrap sample (after holding out the test samples). Instead, I want to obtain statistics about classification accuracy for a specific network.

Another question: in my case, the samples are obtained uniformly from $\mathbb{R}^n$, unlike the typical case when you resample from a finite set of observations. Then, is it still correct to speak of "bootstrapping"?

Thank you!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Bootstrap is a statistical tool help us emulate the process of acquiring new sample set, usually when it is impossible or inconvenient to do so. It resamples from an existing sample set: If you are resampling from the training set to generate the testing sets, it will not be adequate since all these testing cases have been used during your training; if you are resampling from $\mathbb{R}^n$, this is randomly, independently generating test set from the population, which is adequate, but cannot be called bootstrap.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Many thanks for clarifying the meaning of bootstrap. What do you think of the validation scheme I described (i.e., resample only the test set from $\mathbb{R}^n$ and keep the network fixed)? $\endgroup$
    – Nicola Paoletti
    Commented Oct 12, 2017 at 20:44
  • $\begingroup$ It looks good. But a more standard way is to obtain data needed for your model(training/testing) in the first place. For example obtain 10,000 samples. Then divided them into training and testing data. (7,000 for training and 3,000 for testing). From my understanding, what you are doing is training the model used the obtained 7,000 samples and then sampling another 3,000 data from $\mathbb{R}^n$ for testing. This is a little bit odd, but will not cause any problem for validating your model. $\endgroup$
    – X. Zhang
    Commented Oct 12, 2017 at 20:56
  • $\begingroup$ Yes, just to confirm, the workflow would be: 1. Train the model using samples drawn from $\mathbb{R}^n$; 2. Sample N times (e.g. N=1000) the testing set from $\mathbb{R}^n$; 3. Obtain statistics (e.g. mean accuracy and S.E.) by validating the trained model with all test datasets $\endgroup$
    – Nicola Paoletti
    Commented Oct 12, 2017 at 21:57
  • $\begingroup$ This answer does not really say much about the bootstrap. It involves resampling from the original data set with replacement. It is used when parametric assumptions are questionable and it often is used because it has better properties then alternative methods. $\endgroup$
    – Michael R. Chernick
    Commented Oct 12, 2017 at 22:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.