I've got records from patients with different cancers. Naturally, common cancers were registered often, and the cases in this group are then overrepresented and vice versa for the rare cancers.

Let's say that I have:

  • 20 patients cancer 1
  • 160 patients cancer 2
  • 80 patients cancer 3

-> are these strata uneven?

I would like to iteratively sample a set with an equal amount of sufficient cases from each cancer to test on.

However, if I pick 20 cases from each cancer, this won't cover cancer 2, and I will include the same patients from cancer 1 in each iteration. Is this a problem? If so, how would I solve this problem?

If I use sampling with replacement, I do not necessarily pick the same 20 cases from cancer 1, but wouldn't I need more iterations ensure cases from the larger strata are covered enough?

- edit -

I actually need to find features from one particular cancer that are different from the other cancers. I've got thousands of features, couple hundreds of samples from different cancers in different proportions, this is all public data. I'm worried that a standard t-test or some kind of linear model for each feature between cancer of interest vs the rest will be biased because of the distribution of cancer types in 'the rest'.

-> That's why, following the idea described in the methods here, I wanted to subsample the cancers. But this means I would need to iterate over selections of samples, I guess - And that's where the authors of the paper stop providing information.

Then I will need to validate these results in our own samples, and finally we'll need to try and detect these features in specific human samples before we create any kind of model.

So now that this is clear: I think I need a t-test and I think this requires to subsample the cancers. Is this indeed necessary, is this a problem, and if so, how do I solve this?


Removing data you already collected is bad statistical practice, and is usually only done when you are (1) using classification when you should be doing prediction or (2) you are using an improper accuracy scoring rule. The sampling you indicated is guaranteed to make whatever method you use for predictions to fail when tried on new observations. See http://fharrell.com/post/classification and http://fharrell.com/post/class-damage . I'd be interested to know where you learned such a confusing approach. This is getting to be a common statistical misconception.

But I just re-read your question. If you are using a proper probability accuracy scoring rule and are re-sampling just to do an internal validation and not to develop the model, you may be OK to do stratified bootstrapping. There you do the bootstrap individually within each stratum and just combine all the samples.

  • $\begingroup$ I understand that it is a bad approach, but I need a set that has equal amounts of samples from each stratum - because I need to model another biological situation, different from the one where we collect samples at their natural occurrence rate. I got the idea from this article, first paragraph from the methods section, in which authors only give the same description as I did, but they did not provide further information about their reasoning, their methods, nor did they include a discussion about it. $\endgroup$ – MHeydt Jun 4 '18 at 9:00
  • $\begingroup$ Ahh, I see why you provided these links. The ultimate goal is indeed to make a classifier or prediction model. But not by taking a subsample to my liking of this particular data - I will edit my original post. $\endgroup$ – MHeydt Jun 4 '18 at 9:28
  • $\begingroup$ WIthout having time to study that paper, I still lean towards the feeling that subsampling should not be part of the main algorithm but might be used during bootstrap validation. A global statistical model with penalization that is particular to your problem is going to be the best solution. Short of that, some sort of repeated rank ANOVA (e.g., Kruskal-Wallis test) could be contemplated, where you quantify continuous features by how they separate the entire set of possible cancers. $\endgroup$ – Frank Harrell Jun 4 '18 at 11:04
  • $\begingroup$ Okay, I'll look into what I can find out. Thank you for the input! $\endgroup$ – MHeydt Jun 4 '18 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.