1
$\begingroup$

I have performed a time series analysis and I have a dataframe of p-values. I want to performe a False Discovery Rate analysis to account for type I error. I am an R user and I have the following.

library("astsa")

FDR(as.vector(example), qlevel = 0.05)

Example data: https://www.dropbox.com/sh/rjrlgwn48rkcw6d/AACkwOGtfpYvRhco-BU4Lt7ha?dl=0

It is the first time using this statistic and I am quite confused. How can I know if my FDR is good? What is and appropriate value and how can I interpret it?

I would appreciate your help.

$\endgroup$

1 Answer 1

1
$\begingroup$

Reading the code and vignette for FDR from astsa, it seems like it is a lift from this link and they modified it to return the number of rejects.

If we use the code they lifted from, it is a benjamini-hochberg correction:

fdr.basic <- function(pvals,qlevel=0.05){
  n <- length(pvals)
  sorted.pvals <- sort(pvals)
  sort.index <- order(pvals)
  indices <- (1:n)*(sorted.pvals<=qlevel*(1:n)/n)
  num.reject <- max(indices)
  if(num.reject){
    indices <- 1:num.reject
    return(sort(sort.index[indices]))  
  } else{
    return(NULL)
  }
}

And compare it against the commonly used function p.adjust, where we set an FDR of 0.05:

library(astsa)
pv = read.csv("example.csv")
sum(p.adjust(pv$pvals,"BH")<0.05)

FALSE  TRUE 
  450  1650 

length(fdr.basic(pv$pvals,0.05))
[1] 1650

astsa::FDR(pv$pvals)
[1] 1

And you can see the function from astsa gives you something that makes no sense. I suggest you keep to using p.adjust(..,method="BH").

To answer your question, if you set an FDR of 0.05, you expect the proportion of "false discoveries" (rejected null hypotheses that are incorrect rejections) to be 0.05. So in this example if your get 1650 hits with an FDR of 0.05, you can estimate the number of false discoveries to be around 1650*0.05 = 82.5.

It depends on the purpose of your hypothesis testing. If let's say it's for a publication and you want to show you have controlled for multiple testing, the result above is ok, it shows you have an enrichment of so called positive hits.

If you want to come up with a list of good confident or positive hits, then you can go lower with the FDR and see which level suits your purpose best. You can also check out discussions such as this for definition of FDR, FDR for researchers.

$\endgroup$
2
  • $\begingroup$ thank you for your reply and the suggested sources! $\endgroup$
    – geo_dd
    Mar 30, 2020 at 14:09
  • $\begingroup$ you're welcome :) glad it's useful ! $\endgroup$
    – StupidWolf
    Mar 30, 2020 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.