6
$\begingroup$

I have behavioral data for 10 animals that I studied at two different times of day, AM and PM. I am looking to see if there is a significant difference in 8 different behaviors at AM vs. PM. I have two sites (LL and WK), so wanted to look at the differences per site and the combined data. So I performed 24 t-tests and got reasonable p-values, mostly indicating no significant difference between the two times.

I then applied Bonferroni's correction factor, to adjust for the repeated t-tests. However, 22/24 of the adjusted p-values now equal one, which seems odd to me. Any ideas about what went wrong or if these adjusted values are correct?

Here is the code I used to calculate the Bonferroni correction factors:

data2$Bonferroni <- p.adjust(data2$raw.p, method = "bonferroni") 

I've attached the data I used, plus a column including the adjusted p-values. enter image description here

$\endgroup$
15
$\begingroup$

Nothing went wrong. The adjusted p-values are correct. Adjusted $p=1$ simply means no evidence at all for rejecting the null hypothesis.

However

 p.adjust(data2$raw.p, method = "holm")

is always better than the Bonferroni adjustment. Holm's method, which is a step down Bonferroni adjustment, gives the same error rate control as Bonferroni but is more powerful (smaller p-values). As the help page for ?p.adjust says:

There seems no reason to use the unmodified Bonferroni correction because it is dominated by Holm's method, which is also valid under arbitrary assumptions.

For your specific experiment, there is so little evidence of real effects that you won't get any significant results even with Holm's method.

$\endgroup$
  • 1
    $\begingroup$ Not exactly. An adjusted p-value cannot be interpreted quite the same way as a p-value. To say there is "no evidence at all for rejecting the null hypothesis" doesn't make sense. Also, to say Holm's method is "always better" than Bonferroni isn't quite true either, as other discussions have addressed (Holm can't do confidence intervals for example). $\endgroup$ – Bonferroni Jun 14 '17 at 20:21
  • 1
    $\begingroup$ @Bonferroni. Yes, exactly. Holm adjusted p-values are probability bounds so it is correct to call them p-values. $p=1$ has the meaning I attributed to it. Confidence intervals are irrelevant to OP's problem but, nevertheless, one could in principle create CIs to match Holm p-values. $\endgroup$ – Gordon Smyth Jun 23 '17 at 3:30
  • $\begingroup$ Gordon, I notice you still haven't answered the question of what adjusted p-values are the probability of. That's because they aren't the probability of anything. Indeed, saying adjusted p-values have the same interpretation as p-values is simply not true. Adjusted p-values are only "on the same scale as probabilities" because values greater than 1 are arbitrarily "fudged" down to 1. The guy who (literally) wrote the book on adjusted p-values is Peter Westfall, who says the following of Bonferroni-adjusted p-values: "The adjusted p values are not probabilities per se; rather, they are simply c $\endgroup$ – Bonferroni Jun 23 '17 at 22:55
  • $\begingroup$ I have already answered your question, although IMO the unconstructive nature of your posts did not deserve a response. The key term in Peter Westfall's paper is "per se". I am already quite familiar with Prof Westflall work, and the 1997 paper in particular, and I would be happy to split semantic hairs with him. However the semantic war you have tried to provoke is irrelevant to the statistical issues raised here. All previous answers on this page are perfectly compatible with the writings of Prof Westfall and others. $\endgroup$ – Gordon Smyth Jun 25 '17 at 5:50
7
$\begingroup$

Just to add to @gordon-smyth and @student-t 's answer. Another way to look at it is to adjust the $\alpha$-level yourself instead of adjusting the $p$-values via the p.adjust() function. For the Bonferroni correction this is easy enough. If your $\alpha$ level is $0.05$, then you divide this by the number tests, which then is your new Bonferroni adjusted $\alpha$-level.

In your case $0.05/24=0.00208$. As you can see, none of your raw.p's make that cut-off. So your output makes sense.

If you want to use the Holm-Bonferroni method, you can also do this quickly by hand (see here).

$\endgroup$
6
$\begingroup$

Thanks for reading my 1997 JASA paper!

If I had a do-over, I would rephrase my comment that a (single-step) Bonferroni adjusted p-value is not a probability "per se." (And I would no longer use the dreaded "per se." Yecchhhh!)

The Bonferroni adjusted p is in fact an upper bound on the probability that the smallest (random) p-value is smaller than (smaller than or equal to in the discrete case) the given (fixed) p-value, assuming the complete null model describes the randomness. And certainly, 1.0 is an upper bound on any probability.

But the bigger and more important point of my paper is that you can find these adjusted p-values exactly in such a way that accounts for the correlations between the multiple test statistics, assuming the classical linear model. These exact adjusted p-values are in fact probabilities when calculated in single-step fashion; see p. 302 of my JASA paper for the math. (To get the single-step p-values, you need to modify the expression somewhat; see my 1993 Wiley-Interscience book and my SAS book). While I used an enhanced Monte Carlo method to approximate this exact probability, better methods have been developed since; please see Hothorn, T., Bretz, F., and Westfall, P. (2008). Simultaneous Inference in General Parametric Models, Biometrical Journal 50(3), 346–363.

So, single-step adjusted p-values, when computed exactly, are bona fide probabilities.

But, except for the smallest one, step-down adjusted p-values are not bona fide probabilities. They are constructed from bona fide probabilities, but they are not probabilities.

Hope this helps!

$\endgroup$
  • 4
    $\begingroup$ It's always good to see the authors being cited in the discussion giving their inputs here at CV (+1) $\endgroup$ – Firebug Aug 29 '17 at 18:39
2
$\begingroup$

Everything works as expected because Bonferroni could give you adjusted p value greater than one. The R function rounded it off to one because a probability over one makes no sense. This is an example where Bonferroni shows reduction in statistical power.

You may want to try other multiple comparison methods or adjust the significance level.

$\endgroup$
  • 1
    $\begingroup$ Adjusted p-values aren't probabilities. Reducing them to 1 is for aesthetic purposes. $\endgroup$ – Bonferroni Jun 15 '17 at 0:31
  • 2
    $\begingroup$ @Bonferroni. I think you may be getting confused by the different adjustment methods. Bonferroni and Holm adjusted p-values are still p-values (because they are still probabilities) while FDR adjusted p-values are not (because they are rates). Limiting adjusted p-values to 1 is certainly not just for "aesthetic purposes". Adjusted p=1 has an absolute meaning regardless of the adustment method. $\endgroup$ – Gordon Smyth Jun 23 '17 at 3:35
  • $\begingroup$ That's incorrect. Adjusted p-values aren't probabilities (see Westfall, 1997, p. 299), but that's a common misunderstanding. It's not surprising that one would think something called an "adjusted p-value" is a p-value. But what is it you think they are probabilities of, exactly? You'll find there is no sensible answer. A p-value is the probability of, loosely speaking, observing an effect as least as large as the given observed effect, under the null hypothesis. An adjusted p-value obviously cannot mean that--how could there be a 100% probability of observing some effect? $\endgroup$ – Bonferroni Jun 23 '17 at 4:07
  • 1
    $\begingroup$ @Bonferroni. Bonferroni and Holm adjusted p-values provide upper bounds for rejection probabilities under the null hypothesis. They are on the same scale as probabilities, so having a value larger than 1 is meaningless. They have the same interpretation as p-values. In particular adjusted p=1 means inability to reject the null at any significance level; in intuitive terms, this means ``no evidence against the null''. $\endgroup$ – Gordon Smyth Jun 23 '17 at 9:13
  • 1
    $\begingroup$ @Bonferroni. Many statistical tests applied to discrete data can give p-values exactly equal to 1, with or without adjustment for multiple testing. One sided tests provide another example where p-values exactly equal to 1 are common. $\endgroup$ – Gordon Smyth Jun 23 '17 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.