Skip to main content
2 of 5
added 17 characters in body
Gala
  • 8.6k
  • 2
  • 32
  • 44

With 15000 tests, you should indeed have a proportion of rejection close to the nominal error rate and a relatively clean histogram/density plot. If the error level is 1%, 3500 false rejections of the null is definitely a surprising result. My main guess is that successive replicates are not as independent as you thought. I would therefore first try to split each set of 6 observations randomly to assess that.

Also inspect some of these p-values and look at a stripchart or a density plot, just to make sure the histogram is not misleading you and your code is fine. You can also try simulated data, to make sure everything else in your procedure is fine.

One thing you did not specify is the nature of the data and the specific test you are using. With only 3 observations in each group, the sampling distribution of some test statistics (e.g. rank-based tests) can be very unusual.

Gala
  • 8.6k
  • 2
  • 32
  • 44