I'm trying to determine if some particular measurements - in this case taken from a subset of genes of interest (50 genes) - show a significant difference to the rest of the population (15000 genes), by comparing them to a randomly generated subset of 50 genes from within the population of 15000. I present the data for the test genes alongside the random control set in boxplot form, in R, and do a Wilxocon rank sum test to see if there's a significant difference between the two populations:

Example code:

sample50 <- genes[sample(1:nrow(genes), 50, replace = FALSE), ]
wilcox.test(test_genes[, 2], sample50[, 2])

However my problem is that my P-value varies a LOT each time I generate a new set of 50 random values to compare my gene set of interest to (presumably because the population is quite variable, and 50 is a relatively small sample). Does anyone know how I would better determine if my test genes are significantly different? I've been told it might be something like Monte Carlo or Bootstrapping for the random sample but I'm unsure how to do this in my case or how to proceed. Many thanks.


I've been told it might be something like Monte Carlo

What you're doing is Monte Carlo, just not set up in what I think is a sensible way.


Bootstrapping is a related form of resampling. It won't really solve the problem here (at least not of itself -- if you do what's necessary to make bootstrapping work well, lo, a similarly sensibly implemented approach to what you are doing will probably work just fine.)

  1. If you have the whole population, why not compare your 50 to the remainder of the population (the other 1450 values)? What's the difficulty?

  2. If you must sample, why are you not excluding the subset from the population you sample? (there's no point comparing the subset with itself - compare it with everything else; that will answer the substantive question exactly as well but you get the advantage of independence).

  3. If you must sample, why should the sample size be 50? Why not 100? or 200?

  4. Of course the p-value will vary a lot. If the null hypothesis is close to true the distribution of p-values should be close to uniform. If you must sample, and for some reason (I cannot fathom one) you can't take a large sample all at once, sample many times and combine p-values (e.g. via Fisher's method).

  • $\begingroup$ Thanks for the reply. I'll try to clarify: 1&3) I thought (or was told, perhaps incorrectly) that it's better to compare my 50 genes to a random sample of similar size, as opposed to comparing 50 data points to 15000 2) Yes, I could definitely exclude the test values. I was assuming the chance of getting the same 50 genes from a random sample of 15000 was rare (NB not 1500), but it'd be proper to exclude them. May I conclude, then that simply comparing my 50 genes to the rest of the 14950 genes in the population, then doing Wilcoxon, is the best method? Cheers! $\endgroup$ – Michelle Jun 17 '16 at 18:37
  • $\begingroup$ Re 1. Can you explain the reason why it's better to compare 50 with a subset of 50? (Perhaps there's some reason I am missing, in which case it might be important in this case, or perhaps there's something the person who told you that misunderstood.) ... Re 2) sorry about dropping the 0 from 15000; yes, if you sample the larger group down to 50, with those relative sizes it probably won't make enough difference to really bother with so if it's a hassle, you might consider not worrying.. ... "Best" depends on what you want to be best at, and on how good your assumptions are. .... ctd $\endgroup$ – Glen_b -Reinstate Monica Jun 18 '16 at 0:55
  • $\begingroup$ (ctd) ... However, if your 15000 is truly the population of interest (I had assumed it was not), and the 50 is the entirety of the subpopulation you want to compare it with, then you don't need to do a test at all. The point of hypothesis tests is to make inferences about the population. If you truly have the population you're trying to make inferences about you don't calculate p-values, you just compare the quantities of interest directly - if they're different, they're different in the population. When you don't have the population, this is what statistical inference is trying to find out. $\endgroup$ – Glen_b -Reinstate Monica Jun 18 '16 at 0:59
  • $\begingroup$ [I should move some of this up to my answer when I get some more time.] $\endgroup$ – Glen_b -Reinstate Monica Jun 18 '16 at 1:00
  • $\begingroup$ By the way, what kind of thing is being measured for the comparison here? Is it continuous? Is it subject to any kind of detection threshold effects? Is it bounded (e.g. restricted to be non-negative)? If so is the bound attainable? $\endgroup$ – Glen_b -Reinstate Monica Jun 18 '16 at 1:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.