Probability of Confidence Interval Missing: Why Non-smooth w.r.t. Theta? See this: http://ms.mcmaster.ca/peter/s743/poissonalpha.html
The probability that a Confidence Interval will "miss the mark", or fail to contain its target parameter's true value $\theta$, is often a non-smooth function with respect to $\theta$. I vaguely remember seeing this for the Binomial Distribution as well... anyone know why these plots are so wild/misbehaving? 
Also, it seems wrong that $Pr(\theta \notin CI_\theta)>\alpha$ for any $\theta$. What am I missing?
 A: The data from a Poisson distribution (as well as with binomial distribution) is discrete, which makes it impossible to obtain exactly 95 % coverage with arbitrary parameter values. 
One can 'invert' the confidence intervals to define 'acceptance regions': for which possible observations a certain $\mu$ is inside the confidence interval. We require the probability of obtaining an observation within the acceptance region of the true parameter value to be at least $1-\alpha$. For example, when $\mu = 0.2$, the Poisson probabilities $P(X=k)$ for $k=0,\ldots$ are $0.819,0.164,0.016,,\ldots$. The only way to ensure $Pr(0.2 \notin CI_1)\leq 0.05$ is to define confidence intervals so that $0.2$ is within the interval iff the observation is in $\{0,1\}$ (let's call this the acceptance region for $\mu=0.2$), but then the coverage is $0.982$. Following table shows the probability distribution of data, acceptance region and coverage for $\mu=0.2,0.3,0.4$.
\begin{equation}
\begin{array}{c|c|c|c|c}
\mu & P_\mu(X=0) & P_\mu(X=1) & P_\mu(X=2)  & \textrm{AR} & \textrm{Coverage} \\ \hline
0.2 & 0.819 & 0.164 &  0.016 & \{0,1\} & 0.982 \\
0.3 & 0.741 & 0.222 &  0.033 & \{0,1\} & 0.963 \\
0.4 & 0.670 & 0.268 & 0.054  & \{0,1,2\} & 0.992
\end{array}
\end{equation}
When $\mu=0.3$, the acceptance region must be the same as for $\mu=0.2$, but the  probability of covering the parameter value is smaller. This explains why the coverage decreases (1-coverage, as plotted in your link, increases). But then, for $\mu=0.4$, 2 must be included in the acceptance region (equivalently, the confidence interval for $X=2$ must contain $\mu=0.4$, but need not contain $\mu=0.3$) to ensure 95 % coverage. Thus, for confidence intervals defined in this manner, a jump must occur at some point between $\mu=0.3,\mu=0.4$. 
However, the 'exact' confidence intervals defined in the link are designed to fulfill an even stricter requirement: that the probability of missing in either tail must be less than 0.025, and therefore also for $\mu=0.3$, 2 must be in the acceptance region (see the end-points of confidence intervals in Table). Thus, a jump in the green curve occurs at $0.2422$. Note that the green curve never exceeds 0.05 (nor do the blue/red curves showing the missing probabilities in either tail exceed 0.025), thus the coverage requirement is fulfilled exactly.
The second method, 'Pearson' confidence intervals described in the latter part of the link, indeed produces coverages less than $1-\alpha$ for some parameter values. This implies that the method produces only approximate confidence intervals (which is why the first confidence intervals discussed are called exact, while the latter are not). 
A: To summarize the bit from @JuhoKokkala that meant the most to me:


*

*CI's (a range of $\theta$ as a function of $\hat{\theta}$) can be inverted into Acceptance Regions (a range of $\hat{\theta}$ as a function of $\theta$).

*For discrete distributions, the AR's are discrete-valued and thus cannot vary continuously with $\theta$. Thus, discontinuities when a point is added or subtracted from the AR.

*Exact CI's are OK with this. Pearson CI's are only approximately correct (in the limit as $n\rightarrow\infty$).

