How can we possibly know all possible test when applying the Bonferroni correction? I never quite understood the rationale behind the Bonferroni correction. I understand that we want to reduce the FDR, but it seems odd to penalise a test based on the existence of another test, given that we do not even know if we have complete information about all possible test.
As an example, say we perform some test on time series data (let's say, the Mann Kendall test). Let's say we have temperature data about New York from 1900 to 2020. If we perform the test for the full dataset, from 1990 to 2000, that gives us one test. We find this is significant. We could now perform another test, but with a subset, using data from 1950 to 2020. We also find this is significant, but we now apply a Bonferroni correction. We now perform another test, from 1990 to 2020, and now the original test (1900-2020) is no longer significant from the correction. We could continue like this until we have no significant results, even though we haven't changed the raw data.
 A: This is why it's important to have to have a statistical analysis plan, rather than just conducting hypothesis tests on an ad-hoc basis. It's usually good practice to define the hypothesis tests you want to run up front before actually running any of them. You don't need to know the universe of "all possible tests", you just need to know the universe of "tests you're actually running".
It's reasonable to conduct a tiered analysis where different numbers of hypotheses are considered based on domain knowledge. If you have a strong reason to believe there is an effect in some period of time and run a hypothesis test only in that period of time, a result of p<0.05 suggests a true effect. You may then do a more exploratory analysis and run a thousand hypothesis tests over many periods of time, but by the sheer number of tests, it's extremely likely that some of them will have a nominal p<0.05. This new correction factor doesn't invalidate your original finding, since you did run an analysis with one single hypothesis originally. Prior knowledge can be important, as it can lead you to meaningful findings that don't get corrected out by irrelevant statistical tests where you didn't really expect a positive finding anyway.
Suppose I have a 1000 coins and you want to find out if any are biased. If you pick one single coin and get 5 heads in a row, that's rather unexpected and reasonable evidence that the coin is not fair. But if you flip all 1000 coins, it's almost certain that many will produce heads 5 times in a row. This is where honesty in data analysis becomes important, as it would be disingenuous to flip all 1000 coins and then pretend you only flipped the coins that showed p<0.05. In all cases, the raw data remains the same, but how and why you choose to analyze it will affect your interpretation of it.
A: Bonferroni only says: If one has a set of $n$ independent hypothesis tests $\{T_i\}_{i=1}^n$ each with a belonging null hypothesis $H_{0i}$, and one wants the probability of the event, that at least one of those tests has a false discovery, to be less than $\alpha_a$, it is sufficient to make sure that the probability of false discovery of any of those tests is less than $\alpha_a/n$:
$$
\forall_{i=1,\ldots,n} P(T_i(x) = 1 \;|\; H_{0i}) \le \alpha_a/n \quad\Longrightarrow\quad P(\bigvee_{i=1,\ldots,n} T_i(x) = 1 \;| \bigwedge_{i=1,\ldots n} H_{0i}) \le \alpha_a.
$$
Thus, you do not penalise a test based on the existence of another test, but you rather consider a completely different event, the compound event of all the tests together to have no false discovery. This is like when you have two coins: If you want the probability of the first coin to come up tails to be $1/2$, then this will be accomplished by a coin that is balanced ($p_{tails} = p_{heads} = \frac{1}{2}$), and this will not be changed or penalised by what the second coin might come up with. But, of course, if you consider the compound event of both of the two coins to come up tails, and you want the probability of this compound event to be below $1/2$, too, then one way to accomplish this would be to require the coins both to be unbalanced with the probability for tails to be below $1/4$ for both.
Alternatives to Bonferroni: While Bonferroni is clearly correct (and the bounds obtained by dividing by $n$ are rather tight), the problem of the OP still remains: if we are given many tests, how should we combine them into a super-test that doesn't have the disadvantage of the above method that, if there are too many tests, there will never be any discovery, even though there actually would be something to discover? Fortunately, there are alternative procedures. One is the Benjamini–Hochberg (BH) method: In a nutshell, the approach is not to create a single compound hypothesis test. Rather, BH returns those tests $\{Q_i\}_{i=1}^m$, that are most likely to really represent discoveries. For that, BH has a parameter $q$ which we can choose, and then BH guarantees that the ratio of wrong discoveries in the returned test set $\{Q_i\}_{i=1}^m$ will be less than $q$.
