First, let's give a definition of the confidence interval, or, in spaces of dimension greater than one, the confidence region. The definition is a concise version of that given by Jerzy Neyman in his 1937 paper to the Royal Society.
Let the parameter be $\mathfrak{p}$ and the statistic be $\mathfrak{s}$. Each possible parameter value $p$ is associated with an acceptance region $\mathcal{A}(p,\alpha)$ for which $\mathrm{prob}(\mathfrak{s} \in \mathcal{A}(p,\alpha) | \mathfrak{p} = p, \mathcal{I}) = \alpha$, with $\alpha$ being the confidence coefficient, or confidence level (typically 0.95), and $\mathcal{I}$ being the background information which we have to define our probabilities. The confidence region for $\mathfrak{p}$, given $\mathfrak{s} = s$, is then $\mathcal{C}(s,\alpha) = \{p | s \in \mathcal{A}(p,\alpha)\}$.
In other words, the parameter values which form the confidence region are just those whose corresponding $\alpha$-probability region of the sample space contains the statistic.
Now consider that for any possible parameter value $p$:
\begin{align} \int{[p \in \mathcal{C}(s,\alpha)]\:\mathrm{prob}(\mathfrak{s} = s | \mathfrak{p} = p, \mathcal{I})}\:ds &= \int{[s \in \mathcal{A}(p,\alpha)]\:\mathrm{prob}(\mathfrak{s} = s | \mathfrak{p} = p, \mathcal{I})}\:ds \\ &= \alpha \end{align}
where the square brackets are Iverson brackets. This is the key result for a confidence interval or region. It says that the expectation of $[p \in \mathcal{C}(s,\alpha)]$, under the sampling distribution conditional on $p$, is $\alpha$. This result is guaranteed by the construction of the acceptance regions, and moreover it applies to $\mathfrak{p}$, because $\mathfrak{p}$ is a possible parameter value. However, it is not a probability statement about $\mathfrak{p}$, because expectations are not probabilities!
The probability for which that expectation is commonly mistaken is the probability, conditional on $\mathfrak{s} = s$, that the parameter lies in the confidence region:
$$ \mathrm{prob}(\mathfrak{p} \in \mathcal{C}(s,\alpha) | \mathfrak{s} = s, \mathcal{I}) = \frac{\int_{\mathcal{C}(s,\alpha)} \mathrm{prob}(\mathfrak{s} = s | \mathfrak{p} = p, \mathcal{I}) \:\mathrm{prob}(\mathfrak{p} = p | \mathcal{I}) \: dp}{\int \mathrm{prob}(\mathfrak{s} = s | \mathfrak{p} = p, \mathcal{I}) \:\mathrm{prob}(\mathfrak{p} = p | \mathcal{I}) \: dp} $$
This probability reduces to $\alpha$ only for certain combinations of information $\mathcal{I}$ and acceptance regions $\mathcal{A}(p,\alpha)$. For example, if the prior is uniform and the sampling distribution is symmetric in $s$ and $p$ (e.g. a Gaussian with $\mathfrak{p}$$p$ as the mean), then:
\begin{align} \mathrm{prob}(\mathfrak{p} \in \mathcal{C}(s,\alpha) | \mathfrak{s} = s, \mathcal{I}) &= \frac{\int_{\mathcal{C}(s,\alpha)} \mathrm{prob}(\mathfrak{s} = p | \mathfrak{p} = s, \mathcal{I}) \: dp}{\int \mathrm{prob}(\mathfrak{s} = p | \mathfrak{p} = s, \mathcal{I}) \: dp} \\ &= \mathrm{prob}(\mathfrak{s} \in \mathcal{C}(s,\alpha) | \mathfrak{p} = s, \mathcal{I}) \\ &= \mathrm{prob}(s \in \mathcal{A}(\mathfrak{s},\alpha) | \mathfrak{p} = s, \mathcal{I}) \end{align}
If in addition the acceptance regions are such that $s \in \mathcal{A} (\mathfrak{s},\alpha) \iff \mathfrak{s} \in \mathcal{A}(s,\alpha)$, then:
\begin{align} \mathrm{prob}(\mathfrak{p} \in \mathcal{C}(s,\alpha) | \mathfrak{s} = s, \mathcal{I}) &= \mathrm{prob}(\mathfrak{s} \in \mathcal{A}(s,\alpha) | \mathfrak{p} = s, \mathcal{I}) \\ &= \alpha \end{align}
The textbook example of estimating a population mean with a standard confidence interval constructed about a normal statistic is a special case of the preceding assumptions. Therefore the standard 95% confidence interval does contain the mean with probability 0.95; but this correspondence does not generally hold.