Test whether my new method's type I error rate correctly matches chosen alpha

Question

I am testing two different methods for hypothesis testing, bootstrapped likelihood ratio tests, and likelihood ratio tests assuming an asymptotic distribution. I would like to see if the number of rejections at the alpha=5% level is different from 0.05.

Based on this answer to an earlier question (is it correct by the way?), I want to do a binomial test to compare the empirical rejection rate and see whether it is different from 0.05.

However, an alternative solution I am considering is to do a comparative error rate test as described here (and see also here). It does seem the results of the two tests do not fully agree with each other, so I think they are different.

You can compare the two methods as follows. Say my new method has a rejection rate of 5.8% across 5000 samples (so I reject 290). I can do a binomial test in R as follows

stats::binom.test(x=290,n=5000,p=0.05,conf.level=0.95)

    Exact binomial test

data:  290 and 5000
number of successes = 290, number of trials = 5000, p-value = 0.01133
alternative hypothesis: true probability of success is not equal to 0.05
95 percent confidence interval:
 0.05168068 0.06484127
sample estimates:
probability of success 
                 0.058 

And thus I can reject the idea that my test has the proper rejection rate. For the second method, we can use this online calculator suggested in the other post, by saying Sample Size 1 = 5000, Percentage Response 1 = 5.8, Sample Size 2 =5000, Percentage Response 2 = 5. And we get the following results: Comparative Error :0.89, Difference :0.8, Significance :No.

My guess is that the binomial test correctly tells me what I really care about, whether or not the rejection rate of my new method is correct or not in the sense of whether it matches my alpha value. However, I'm comparing a few different methods, and perhaps the "comparative error" measurement could be useful in some way for comparing methods?

Answer 1

Empirical validation of null-rejection methods

tldr:

• in null-rejection testing, the empirical rate is typically different than the nominal rate (due to approximations), but it only matters if it is meaningfully different.
• if you think the rate is $5%$ but it's actually $4.9%$, that difference doesn't matter. If it's $10%$, that's very different.
• running experiments on artificial data are a good way to check how the empirical rate behaves.

1. The setup

Let's consider the following situation. We are doing some science 🧪, and, in our experiment, we want to compare two classes. We have:

• a null hypothesis $H_0$: the two classes are identical
• the alternative $H_1$: the two classes are different
  • Note, even in the simplest case $H_1$ is an ensemble! In complex examples, $H_0$ is also an ensemble.
• a number of possibles tests. These are methods which label an input dataset as having a significant difference $S_1$ or unsignificant difference $S_0$

For example, we might want to see whether two groups of values have the same distribution (null hypothesis) or not. To do so, we might test whether they have the same empirical mean or empirical median or empirical variance, etc.

We want to validate the general worth of these various tests for a particular problem.

2. The measures

In a frequentist framework, we have two canonical measures associated to this problem.

• the false positive rate (FPR): for a given null model (ie, a given element of $H_0$), the rate at which the algorithm produces a $S_1$ result.
• the false negative rate (FNR): for a given non-null model (ie, a given element of $H_1$), the rate at which the algorithm produces a $S_0$ result.

Typically, the FPR is exactly the same throughout $H_0$, but that is not necessary. Technically, the usual guarantee is that the false positive rate is, at most, some $\alpha$ close to 0 (typically 0.05 or 0.01 or 0.001). $\alpha$ is sometimes called the level of the test.

In contrast, typically, the FNR varies as we vary the non-null model: it is a function of the specific non-null model. This is understandable: some cases are more easy to discriminate from a null while some are more ambiguous. In our example, if we are using the empirical mean, it is easier to reject the null if the difference of the true means is 100 instead of 1 (keeping the variances constant). This function of non-null model to FNR is called the power function of the test.

The FPR and FNR are linked. Typically, the rate of classification is continuous as we vary the model. For models that are thus almost-null (there is an effect but almost imperceptible, ie: the two means are almost identical), the FNR is $1 - \alpha$. If we go further from the null, then the FNR is going to go down in some method-specific way.

3. Empirical validation

Critically, when we design a test to have a given FPR $\alpha$, there are typically assumptions being made about the ensemble $H_0$, and the ensemble $H_1$. For example, we might make assumptions about the data distribution, dependence structure, etc. We might also make assumptions about the data being large-enough (and the distributions being well-behaved) to justify asymptotic approximations of key intermediate quantities. These assumptions might not be valid for the kind of experiments we are performing in a specific domain.

We thus might want to check that, in a number of experiments with controlled generative model (ie: data which we generate ourselves from a known model) the theoretical properties of the method (derived from the analysis) match the empirical properties (which we can observe in the epxeriments). This is the approach of Caron, 2017

For example, we can choose a specific model in $H_0$, generate some large number $m$ of datasets from that model, and measure the empirical rate of $S_1$. We can then compare that empirical rate to the theoretical rate.

Critically, we are not interested in whether the difference is statistically significant. We should take $m$ large enough so that the error bars on the empirical rate are much smaller than the theoretical rate $\alpha$. As a consequence, we can consider that the empirical rate is constant. In the results from the Caron paper, almost all their empirical rates are statistically different from the nominal rate.

What we should care about is whether the difference is meaningful. Indeed, the FPR being $4.99%$ is not far enough from $5%$ to care. Caron gives the following rule: he highlights all cases where the theoretical rate is $5%$ but the empirical rate is lower than $2.5%$ or higher than $7.5%$. The details of that boundary are subjective, but that sounds about right to me.

If we want to be absolutely thorough, we should also investigate the FNR of the methods in $H_1$. However, due to the link of the FRP to the $1-FNR$ for almost-null models, this is not necessarily needed.

Note, like all experiments, the goal should be to extract meaning and intuition from the results. This should require some careful thinking about how to explore the ensemble $H_0$ and $H_1$ and summarize the results.

Note that, typically, we should expect the bootstrap tests to be succesful in such experiments because they are designed to minimize assumptions about the data.

Answer 2

This is not an answer about statistical methods, but only to point out that it is common practice in the literature to not actually test whether a rejection rate is significantly different than the alpha value, but instead simply eye ball it.

For example Caron 2019 simply says that rejection rates outside of [0.025, 0.075] will be considered bad.

Similarly, Nestler Salditt 2024, PETRINOVICH and HARDYCK 1969, and Cooper et. al. 2016 all seem to eye ball it, though I am not 100% sure.

I've only found one paper so far that assessed the rejection rates as being significantly different than the alpha value, but I've lost the link to it.