3
$\begingroup$

I am using biological / microarray data. For example, one of my datasets has 50 samples, and 1000 gene attributes. They have 2 labels, Normal and Disease. I usually use a machine learning method such as SVM to classify them. In my field, 10-fold cross-validation is used very frequently, but because the sample size is small, robust results are difficult to acquire. So I usually use resampling to test. For example, make 100 subsets each of which contains 40 samples (80% of 50 samples) randomly from 50 samples.

Recently, I learned the concept of the bootstrap, which means sampling with replacement. In my resampling example, there will be at least 60% (30) of overlap for 2 sets. So, I think bootstrap is the same concept as resampling, is it right understanding?

And, I am using moderate ratio (80%) for resampling, is there any rule to decide it? There are some rules like 632+ in bootstrap, but I think they did not suggest the ratio of resampling.

$\endgroup$
  • $\begingroup$ This is a bit hard to follow. Can you say more about your data? What are your variables? Are you comparing two groups (sets)? What does it mean to have ___% overlap? $\endgroup$ – gung - Reinstate Monica Jan 11 '16 at 17:05
  • $\begingroup$ @gung Sorry for poor explanation. I edited my question now. $\endgroup$ – z991 Jan 11 '16 at 17:16
  • $\begingroup$ The bootstrap is a kind of resampling, it's not the same thing as resampling, since there are resampling methods that are not bootstrapping (e.g. randomization tests are also resampling methods). $\endgroup$ – Glen_b Jan 12 '16 at 6:51
  • $\begingroup$ @Geln_b Thank you for comment. In my case, I took 40 random sample from 50 training samples 2 times, then there will be overlap of 30~40 samples between 2 subsets. In other words, the difference between 2 subset are 0~10. The, in this case is it the same concept as 'replacement' of bootstrap? $\endgroup$ – z991 Jan 12 '16 at 7:55
7
$\begingroup$

The immediate answer is that these are two very different techniques, with very different purposes. To make things clear let me explain the techniques one by one, and then talk about them in this context.

Resampling for Cross Validation

When you use cross validation (CV), you are trying to make sure that your model is capturing the signal in your data, and not the noise -- in other words, the part that's similar among all the data (signal), and not the part that's unique to just the data you have (noise).

If you're capturing the signal, then you should be able to predict that signal in the hold-out data, even though you didn't use it in the original model, and that should give you good classification accuracy. If you're capturing the noise, your model will not be able to predict anything in the hold-out data, because the noise is unique to the data you used in the original model. As a result, you get bad classification accuracy.

The reason to HAVE to make sure your training data and testing data are separate is that if you put the same noise in both the training and testing data, you'll think you're predicting signal, when in fact, you're just using one unique data point to predict itself.

Resampling for a Bootstrap

In general, the bootstrap is not used for cross-validation (I have never seen it done). It is used to calculate confidence intervals. You sample from your original data with replacement, reestimate your quantities of interest with the resampled data, and then see how much those estimates vary -- that variance can be used to calculate a confidence interval.

Bootstrapping also involves resampling, but the reason it does it differently is that it has a different goal in mind. To quote from ttnphns's helpful comment, "bootstrapping simulates/approximates the asymptotic interval estimation (under assumption that the sample and the population distributions are isomorphic) of the infinite/large population and that the only sampling strategy which may correctly link a finite set with an infinite one is the replacement."

Meaning, loosely, if your data is representative of your underlying distribution, then you can use a bootstrap method to infer that underlying distribution, but doing that requires sampling with replacement.

Comparing them in this context

You mention "your sampling" a lot without it being clear what you're referring to, so first, it's important to make sure that, however you're choosing your train and test sets (for a single fold), there is NO overlap between them.

Second, "bootstrap" is not the same as "resampling," or "sampling with replacement," in general. Many things have sampling with replacement. Just because there is sampling with replacement does not mean something is a bootstrap.

Here though, there IS something you can do in this context (that might be okay to call a "bootstrap"), and that is with your prediction accuracy. You also never mention your prediction accuracy, so I'm guessing somewhat that that's what you mean, but it's not a bad idea, and here's what it is and why it works (I think, and whether or not it's really a "bootstrap").

You say your "results" are not robust because you don't have much data. By "results," I'm going to assume you mean "cross-validation accuracy." And if there's a lot of variance in your calculated CV-accuracy, then I see why only using a small number of folds isn't going to be enough -- you want a clearer picture of the full distribution of what your hold-out accuracy might be. And to do this, you're randomly choosing 40 data points, and predicting on 10 -- randomly choosing 40 new data points, and predicting on the remaining 10. Each time you get a new accuracy.

I think it is okay to say that you are using a bootstrap method, in order to find a confidence interval for your cross-validation accuracy (but I'm also not entirely certain). You are randomly sampling points from a "cross-validation space." The space is complicated, and I'm not sure what implications that might have -- but you are sampling the points in it randomly, and I think the distribution you get back, of possible accuracies, is meaningful. (ASSUMING, always, that your 50 data points really are representative of your underlying distribution. How good an assumption THAT is though, I have NO idea -- that's your job.) Ultimately, this is not really different from n-fold CV, where a single point can be in the training set as many as $N-1$ different times. You're just taking a random subset of those results, but if it's random, I think you're okay.

(And if you're worried, you could also cache (save) the indices of the samples you've trained on, so that you don't repeat observations. But, you should be safe -- 50 choose 10 is ten billion.)

Last, re: choosing a split (like 80%,) there is no solid rule. But the whole point is to make sure you're not over-fitting, and because cross-validation can help test for over-fitting, you can try a number of different splits (between 50%-90%, say) and see which of those gives the best testing accuracy. (Just make sure you're not over-fitting this number, too!) Here, because you want an lot of points, I would start with leave-one-out-cross-validation (LOOCV) and compare it to the 80% you're using now.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1 for the nice overview of the two concepts. I'd just suggest to expand your bootstrap paragraph by saying something like that bootstrapping simulates/approximates the asymptotic interval estimation (under assumption that the sample and the population distributions are isomorphic) of the infinite/large population and that the only sampling strategy which may correctly link a finite set with an infinite one is the replacement. $\endgroup$ – ttnphns Jan 11 '16 at 20:22
  • $\begingroup$ Thanks - I'll do that when I get the chance (aided by your comment, which I would not have known to include). $\endgroup$ – one_observation Jan 11 '16 at 22:02
  • $\begingroup$ Thank you very much! It was a very nice and clear explanation for me. But one thing I did not understand is the difference between bootstrap and my sampling. I am sorry that I did not mention about test set, but there was separated test set of 10 samples in my data. So, my method was taking 40 random sample from 50 train sample, and test with test dataset. And take another 40 random sample, which have max 40(if the same samples are picked) and min 30 overlaps with previous data. Then, is it the same concept with bootstrap? And is there no rule for selecting max and min overlaps? $\endgroup$ – z991 Jan 12 '16 at 4:59
  • $\begingroup$ Ok -- I think I finally get it, but there are a lot of reasons this is very difficult to follow. $\endgroup$ – one_observation Jan 12 '16 at 20:01
  • $\begingroup$ @Sophologist Thank you very much for your great explanation! Now I could understand the difference of them and my problem. $\endgroup$ – z991 Jan 13 '16 at 6:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.