Precise definition of the confidence interval - sources? I always thought that the CI is defined as the set of parameter values corresponding to hypotheses which we can retain. But I struggle to find a source for this definition. Is it not correct?
Most sources (including Wikipedia) seem to define the 95% CI by the requirement that for a fixed true value of the parameter, there's 95% probability that the CI resulting from a random experiment will cover it. Some sources give wrong and/or murky definitions (or definitions specific to for example the t-test), but the Wikipedia definition seems to be the consensus.
I have two problems with this definition.
First, by this definition the CI simply wouldn't exist in many cases. For example, consider the throw of a single coin with a true head probability of, say, 0.42. There's only two possible outcomes of our experiment. If CI(X=head) contains 0.42 but CI(X=tail) does not, then our CI() function satisfies the definition if a 42% confidence interval (at least for the true value theta=0.42), but there's no way the CI function could ever give you a coverage probability of 0.95.
Second, it is not obvious that the Wikipedia definition uniquely defines anything. What would be wrong with the following stochastic process? With 95% probability, return ]-inf;Inf[. With 5% probability, return the empty set
OK, maybe one can prove that the above silly definition creates a CI() function that will never be a function of a minimal sufficient statistic for the estimation of the parameter of interest, and then deem it illegal for that reason. But it strikes me as wrong to provide a "definition" of the CI by requiring to have some property, without making it obvious that this property uniquely defines the CI.
 A: The theoretical interpretation and practical use of confidence intervals (CIs) can lead to confusion because people forget what probabilities mean. Terminology such as "confidence" and "coverage probability," perhaps originally intended to make CIs easier to understand, may have had the opposite effect.
Reminders about 'probability'. Suppose you are told the $Z \sim \mathsf{Norm}(0,1)$ and then you consult a printed normal table or use software to find that
$P(Z > 1) \approx  0.1587.$ 
According to the frequentist definition
of probability this imagines that you can observe values $Z_1, Z_2, Z_3, \dots $ repeatedly over time. Some of these $Z_i$ will lie above 1 and some will not. The probability statement says that over the long run you will see a $Z_i > 1$ on about $16 \%$  of the observations, and the rest of the time not.
Few people would ask for the "coverage probability" of the event
$\{Z > 1\}.$ Perhaps fewer would say that once you observed a $Z_i,$ the expression  $P(Z > 1)$ becomes meaningless because either your $Z_i >1$ or it isn't, and there's "no probability about it," so suddenly you'd now need to say you have about $16\%$ "confidence" in the event $\{Z > 1\}.$ 
A confidence interval. By contrast, suppose you are sampling from some normal population.
You know that $\sigma = 2,$ but you don't know $\mu.$ You get to observe a sequence of observations $X_1, X_2, X_3, \dots, X_n$ from
the population and you compute their mean $\bar X.$ 
You know that
$$0.95 = \left(-1.96 \le \frac{\bar X-\mu}{\sigma/\sqrt{n}} = Z \le 1.96\right)\\
= P\left(\bar X - 1.96\frac{\sigma}{\sqrt{n}} \le \mu \le \bar X - 1.96\frac{\sigma}{\sqrt{n}}\right).$$ 
And you say $``$Aha! I can use $\left(\bar X - 1.96\frac{\sigma}{\sqrt{n}}, \: \bar X - 1.96\frac{\sigma}{\sqrt{n}}\right)$
as an interval estimate of $\mu."$
Quibbles. Then somebody asks what you mean by that, and you say that, over the long run, on 95% of such $n$-sample experiments from this population the true value of $\mu$ will lie between these endpoints. But now some people feel free to say, you can't use
the word probability in connection with this interval. Once
you've computed $\bar X$ for this experiment, there's no "probability" about it. Either the endpoints include $\mu$ or they don't. 
A compromise is reached, everybody agrees to call this thing a confidence interval. Even so, the same people might ask you what the coverage probability of the interval is. 
Consulting. If you've just finished a consulting assignment and researchers ask you what the CI in your report means, you could give them
the story about an infinite string of possible repetitions of their experiment, but they're focused on the one just concluded.
Then you might say the CI has a 95% coverage probability. 
And they'll say something like, "So there's a 95\% chance our $\mu$ is in that interval." You could say, "Something like that." Or realizing that, absent divine revelation, the exact true $\mu$ will never be known, even though you do have an exact number for $\bar X,$ you could gingerly cross an invisible Bayesian line and just say "Yes."
Duality with testing. When the CI arises from a pivotal quantity, as above, pivoting from and inequality were constants bound a random variable to one where random variables bound a parameter, then you can usually find a "matching" hypothesis testing situation such as $H_0: \mu = \mu_0$ against
$H_a: \mu \ne \mu_0.$ 
Then it's true that the CI is an interval of non-rejectable values $\mu_0.$ 
Recognized probabilities for tests. Now define the test statistic as $Z = \frac{\bar X - \mu_0}{\sigma/\sqrt{n}}$ and agree to reject $H_0: \mu = \mu_0$ if $|Z| \ge 1.96.$ Then you can say, $``$Given that $H_0$ is true,
$P(\mathrm{Reject}\,|\,\mu_0) = \alpha = 5\%."$ And you can say, $``$Given that $H_a$ is true with $\mu = \mu_a \ne \mu_0,$ then
the power of the test against alternative $\mu_a$ is
$P(\mathrm{Reject}\,|\,\mu_1)."$ In both of these cases, you start by assuming a parameter value, so the word probability seems generally acceptable.  
A: I share your sentiments about the clumsiness of the definition of a confidence interval, and I think historically it was first defined in the way you suggested, i.e., "the set of parameter values corresponding to hypotheses which we can retain", although I may be wrong. 
As for the relationship between your definition and the Wikipedia definition, see, e.g., my recent answer to a related question here. For a more authoritative reference, you can consult Cox and Hinkley (1974), I believe. However, since there are multiple ways to define a test, there are therefore multiple ways to define confidence intervals and so CIs are not unique. Your example is a good one which illustrate how you can have  pointless CIs based on utterly useless procedures. 
