How to calculate the confidence interval of expected number of rejctions If I repeat a test (e.g. a Chi-square test) 1000 times by Monte Carlo simulation, I would expect 50 rejections for $\alpha=0.05$.
But how do I calculate the confidence interval of the number of rejections at $\alpha=0.05$ level?
 A: There are actually a few problems with your question.


*

*You say that you expect 50 times of rejections for $\alpha=0.05$. But you only expect that under the null hypothesis (and assuming that your statistic allows you to exhaust your $\alpha$-level budget, i.e. that the corresponding p-value is uniform under the null)! If you draw under the alternative, depending on the power of your test, you should get a lot more rejections.

*You ask for a confidence interval for the number of rejections. This does not make sense! You construct confidence intervals for parameters which you want to estimate. But here there is no parameter you are estimating. Instead you are interested in the distribution of the number of rejections. In particular, it seems to me like you are interested in the quantiles of the distribution of the number of rejections.
And now having hopefully cleared up any confusion you had about these terms, the answer to your (intended) question is easy: Assume you do $B$ Monte carlo replications under the null hypothesis of your test at level $\alpha$. Well, you just described a Binomial experiment, i.e. the number of rejections follows a Binomial Distribution:
$$ R \sim \text{Binomial}(B,\alpha) $$
From this you can calculate quantiles, variance etc. You also get the expected value you provided in your post ($\alpha B$).
UPDATE: So in the paper (which I did not read; but I will take Jessie's word for it) they say that for $B=1000$, $\alpha=0.05$, one gets a 95% confidence interval of $[36,64]$ for the number of rejections. As I wrote above, this is not a confidence interval. But what they actually mean is that they are looking for an interval (not confidence interval) such that with a probability of 95% the number of rejections will lie in this interval. So there are a few ways we can do this, but we could for example just exclude very high values and very low values in a symmetric way. So we could look for say the 97.5% quantile and the 2.5% quantile and remove these from our interval. In R:
>qbinom(.975, 1000, 0.05)
[1] 64
>qbinom(.025, 1000, 0.05)
[1] 37

So I actually get the interval $[37,64]$ and I do not know why they got $[36,64]$. We can check that indeed there's a $\geq .95$ probability for the number of rejections to land in this interval:
> pbinom(64, 1000,0.05) - pbinom(36, 1000, 0.05)
[1] 0.9580953

Their interval would also be valid, though a bit too wide
> pbinom(64, 1000,0.05) - pbinom(35, 1000, 0.05)
[1] 0.9650301

