Why is this a problem? : Error: "the researcher is assuming an alpha level below the nominal" (p < 0.05) I'm trying to understand an article about significance testing and ANOVA, where Monte Carlo simulations were run. The final sentence of this paragraph (in bold) makes absolutely no sense to me.

"Type I error is the probability of rejecting a null hypothesis when it is actually true. The robustness of a statistical test can be evaluated via Monte Carlo simulation techniques, and in order to ensure the comparability of results from Monte Carlo studies a standard criterion to assess robustness must be established. Bradley’s (1978) liberal criterion is considered the most appropriate (e.g.,Keselman, Algina, Kowalchuk, & Wolfinger, 1999;Kowalchuk, Keselman, Algina, & Wolfinger, 2004). According to this criterion, a statistical test is considered robust if the empirical Type I error rate is between .025 and .075 for a nominal alpha level of .05. When the rate is above .075 the test is considered liberal, increasing the risk of declaring mean differences that do not exist. When the rate is below .025 the test is considered conservative, such that the researcher is assuming an alpha level below the nominal" (Blanca et al., 2018).

Throughout the article, rates below .025 or above .075 are considered "non-robust."  Robustness, a binary variable, is what's used in all the chi-squares testing the article talks about. The point of the article to assist researchers in determining whether they can trust their F-test, based on differences in group variance and group sample size.
My question is, why is it considered a failure here to have an actual significance level of 0.01 when you did a wide range of simulations at a significance level of 0.05? And how is this being attributed to "the researcher?" This is simulated data, where the differences in variance and sample size are being adjusted, while maintaining normality.
I'm looking for an explanation of the last sentence in the above paragraph.
Reference: Blanca, M. J., Alarcón, R., Arnau, J., Bono, R., & Bendayan, R. (2018). Effect of variance ratio on ANOVA robustness: Might 1.5 be the limit? Behavior Research Methods, 50(3), 937–962.
(This particular article is open-access through Springer.)
 A: The Type I error rate is the probability of rejecting a null hypothesis when it is actually true. For a test with a 0.05 Type I error rate, we should be able to generate many datasets under the null hypothesis, and see that we reject the null 5% of the time. If we see that our Monte Carlo simulation results in the null being rejected only 1% of the time, however, our test is actually more powerful than we thought - only 1% of our rejected nulls are false, not 5%. Conversely, if we reject 10% of our nulls, our test is not as good as assumed, since our empirical Type I error rate is higher than the nominal alpha level of 0.05. The authors suggest here that an "acceptable" window around a nominal 5% Type I error rate is within a true error rate of 2.5% and 7.5%.
A statistical test is not considered "robust" if its empirical Type I error rate does not match up with its stated alpha level - the test isn't doing what it says "on the tin", and is actually rejecting null hypotheses more or less frequently than we'd expect. It's not the worst thing to have a test more powerful than you expect, but it is more conservative and will fail to find true differences more often than you're expecting. If you want a 5% Type I error rate, you shouldn't use a test with a 1% Type I error rate - if you do, you're implicitly tightening your significance threshold. The opposite is even worse, if your test is less powerful than you expect, you might be falsely rejecting nulls more often than you realize - believing a test has a 1% error rate when it's actually 5% can lead to spurious rejections of the null.
A: When you declare $\alpha = 0.05$, you are saying that you are okay with a type $1$ error rate of $5\%$.
When the realized type 1 error rate is below the nominal, it means that you have a type $1$ error rate less than $5\%$.
While it might sound like great news to be making fewer mistakes, you had already declared that you were okay with making a certain number of mistakes. Consequently, you can raise $\alpha$ until you realize that tolerable error rate of $5\%$.
By raising your $\alpha$-level, you increase your power to detect differences, and, despite the $\alpha>0.05$, your observed type $1$ error rate is at that tolerable limit of $5\%$.
Conversely, if you observe a type $1$ error rate higher than $\alpha$, you are accepting more type $1$ error, which, by setting $\alpha$, you had decided was unacceptable. In order to get back to the acceptable type $1$ error rate, you must lower $\alpha$, and this decreases power to detect differences.
Let's do an R example to see how to tune $\alpha$. I will use a Wilcoxon Mann-Whitney U test to examine for a difference in means between $Beta(1/2, 1/2)$ and $N(1/2, 1)$, which have the same mean of $1/2$. The Wilcoxon Mann-Whitney U test is not quite a test of mean equality, but it often gets used that way.
set.seed(2021)
alpha <- 0.05
N <- 100
B <- 1000
ps <- rep(NA, B)
for (i in 1:B){
    x <- rnorm(N, 1/2, 1)
    y <- rbeta(N, 1/2, 1/2)
    ps[i] <- wilcox.test(x, y)$p.value
}
ecdf(ps)(alpha) # 0.076

That final line says that the type $1$ error rate of this test is $7.6\%$, when you had decided that you wanted an error rate of $5\%$. To determine the $\alpha$ we should use to get the desired $5\%$ type $1$ error rate, use the quantile function.
quantile(ps, 0.05) # 0.0336058281017968

So we should pick $\alpha=0.0336$ to get the desired type $1$ error rate.
set.seed(2021)
alpha <- 0.0336
N <- 100
B <- 1000
ps <- rep(NA, B)
for (i in 1:B){
    x <- rnorm(N, 1/2, 1)
    y <- rbeta(N, 1/2, 1/2)
    ps[i] <- wilcox.test(x, y)$p.value
}
ecdf(ps)(alpha) # 0.05

This change in alpha, however, decreases our power to reject a false null hypothesis, such as $Beta(1/2, 1/2)$ and $N(1/3, 1)$.
set.seed(2021)
alpha <- 0.0336
N <- 100
B <- 1000
ps <- rep(NA, B)
for (i in 1:B){
    x <- rnorm(N, 1/3, 1)
    y <- rbeta(N, 1/2, 1/2)
    ps[i] <- wilcox.test(x, y)$p.value
}
ecdf(ps)(0.05) # 0.358
ecdf(ps)(0.0336) # 0.295

When we have to lower $\alpha$, we lose some power to detect this difference, dropping from $35.8\%$ to $29.5\%$.
ANALOGY
The speedometer in my car always gives a speed that is higher than the speed I am traveling. Consequently, if I want to drive on a road with a speed limit of $60$, I should push the speedometer up above $60$. This has the benefit of getting me to my destination sooner than I would if I drove at a speed that resulted in a reading of $60$. However, if my car displays a speed lower than I am traveling, I need to drive at a speed that displays as lower than $60$. Compared to my drive when the speedometer displayed $60$, this will take longer. However, I am traveling $60$, which is the maximum speed allowed before I will get a speeding ticket.
A: The statement in the article, "the researcher is assuming an alpha level below the nominal," is invalid. (Assuming the statement was valid was causing me to have problems interpreting the article.)
Nothing is being said about the importance of a Type I error vs. a Type II error. What is acceptable in sociological research may not be acceptable in pharmaceutical research.  The power of a test is a research choice, but again, that isn't addressed in the article.
Initially, I thought the author was only reporting robust vs. non-robust results in the tables, and given that p-values below .025 were treated as failures (non-robust), just like values above .075 are, it appeared the tables would be practically worthless to people interested in knowing how variance and sample size differences could cause Type I errors - which is what the article says the article is about.
Fortunately, there are tables in the Appendix which show the actual F-test values for all the scenarios, which will be very useful. You can match up your ANOVA scenario, and watch how the F-test results are affected by increases in the variance ratio.
