2
$\begingroup$

I am trying to understand how FDR is calculated and what to understand from it. I selected R as language so the example is in R

Lets look at an example of 10 samples

df<- structure(list(Names = structure(c(1L, 3L, 4L, 5L, 6L, 7L, 8L, 
9L, 10L, 2L), .Label = c("Sample1", "Sample10", "Sample2", "Sample3", 
"Sample4", "Sample5", "Sample6", "Sample7", "Sample8", "Sample9"
), class = "factor"), p_value_1 = c(0.01, 0.02, 0.03, 0.05, 0.013, 
0.014, 0.019, 0.35, 0.5, 0.63)), class = "data.frame", row.names = c(NA, 
-10L))

To calculate FDR values , one should use adjusted-p_value which I think is influenced by the size of the sample , how ? I don't know. One can calculate it based on different method for example

FDR <- p.adjust(df$p_value_1, method="fdr")

I came across a study which I cannot understand. is the following formula FDR?

enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

How to do the math
To adjust for multiple comparisons using the Benjamini-Hochberg procedure for controlling the False Discovery Rate (FDR) of $\alpha$ on $m$ different tests ($m$ is not sample size):

  1. Compute the p-value for each test.
  2. Order the p-values from largest to smallest.
  3. In the first (ordered) test ($i=1$), compare the -value to $\frac{\alpha\times(m+1–1)}{m}$ (if you are using, say, $p=P(|T|>|t|)$ for two-tailed tests, if using $p=P(T>|t|)$, then replace $\alpha$ with $\alpha/2$).
  4. In the second test ($i=2$), compare the -value to $\frac{\alpha\times(m+1–2)}{m}$.
  5. In the third test ($i=3$), compare the -value to $\frac{\alpha\times(m+1–3)}{m}$.
  6. In the ith test, compare the -value to $\frac{\alpha\times(m+1–i)}{m}$.
  7. Using Benjamini & Hochberg’s method, we reject all tests including and following the first test for which we reject the null hypothesis.

Adjusted p-values simply invert the above math to transform p, instead of alter the rejection threshold. So reject if $p \le \frac{\alpha (m+1–i)}{m}$ becomes reject if $\frac{mp}{(m+1–i)} \le \alpha$

Important things to note about adjusted p values

  • Adjusted p-values can be computed as $>1$ which is incoherent, since p-values are supposed to be probabilities. These are typically reported simply as $1$.
  • One cannot determine whether any given test is significant or not simply by looking at the adjusted p-values alone, since these decisions also depend on the ordering of the unadjusted p-values.
  • p.adjust() for some reason has a arithmetic error relative to Benjamini & Hochberg: the first two adjusted p values it reports are always given the same value in my experience.
$\endgroup$
6
  • $\begingroup$ I want to accept and like your answer, however, I have some questions regarding your answer. 1- can you give an example how to calculate it? second what about the definition of null hypothesis? can you please explain that ? Please look at this manuscript jstor.org/stable/2346101?seq=1#page_scan_tab_contents $\endgroup$
    – Learner
    Mar 28, 2019 at 21:51
  • $\begingroup$ @Learner Hi Learner... I will be happy to add a worked example, but it may be a few hours. I am indeed familiar with the article you cite, and have read it a few times. I am not sure I follow you about null hypotheses? If you have a question about what they are you should ask a new question! :) $\endgroup$
    – Alexis
    Mar 28, 2019 at 22:34
  • $\begingroup$ can you at least explain what is T, what is m? $\endgroup$
    – Learner
    Apr 3, 2019 at 2:41
  • $\begingroup$ $m$ is the number of tests. $P(T>|t|)$ literally "the probability of observing some value of a t test statistic greater than or equal to the absolute value of $t$." $P(|T|>|t|)$ literally "the probability of observing some value of a t test statistic greater than or equal to the absolute value of $t$ or less than or equal to negative one time the absolute value of $t$." But t test statistics are just an example. FDR works for any group of hypothesis tests. $\endgroup$
    – Alexis
    Apr 3, 2019 at 14:39
  • $\begingroup$ how can I calculate the probabilities with hand? you make me more confused than I should be!!!!!! Can you give an example of 5 p values ? lets say I have a group (control) and a group of treated of 5 patients, I obtain 5 p values. Now how can I calculate the FDR with hand ? $\endgroup$
    – Learner
    Apr 3, 2019 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.