1
$\begingroup$

I would like to apply a null hypothesis significance test that requires iid data (specifically the Pettitt test for change-point detection) to multiple time series in different locations. These time series typically exhibit significant autocorrelation to some lag. My understanding is that both autocorrelation and multiple testing can lead to inflation of the Type I error rate, and hence an over-rejection of the null hypothesis.

To deal with multiple testing, one can control the false discovery rate (FDR). For autocorrelated data, one might apply prewhitening methods to the time series, or perhaps use block bootstrapping.

My conceptual understanding of these kinds of treatments is that they effectively try to apply a stricter threshold of significance than one would use without accounting for multiple testing or spatial correlation (e.g. simply taking $p<0.05$ as indicating significance). Practically, when I have applied FDR to my data, this results in a much smaller p-value threshold for indicating significance, while block bootstrapping typically results in larger p-values than those generated using non-block bootstrapping.

I have difficulty in understanding whether both multiple testing and autocorrelation always need be accounted for when significance testing, or whether judicious treatment of either one of them would suffice. For example, as FDR controls the number of false rejections, does it implicitly account for the effects of autocorrelation (or any other effect that would lead to an inflated Type I error rate)?

From a practical (i.e. programming) point of view, it would be very beneficial for me to use FDR, but safely ignore autocorrelation.

My question is: Given I am applying hypothesis tests to autocorrelated data for numerous time series in space, should I account for both multiple testing and autocorrelation, and what is the logic behind this?

Note: I have some grasp of statistics, and often struggle more with the concepts than math, but I am really trying to learn more and become more statistically competent. Links to journal articles/books that can help me understand the problem would be much appreciated.

$\endgroup$

1 Answer 1

0
$\begingroup$

Short answer is no.

Take it in two steps. First, ensure that the tests are valid, irrespective of any multiple testing correction. In your case, that means accounting for autocorrelation. Then you can apply multiple testing corrections like Bonferroni and standard fdr to the results.

To take it a step further and improve power, there are modifications of Bonferroni and standard fdr that incorporate dependence structures.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.