A confidence interval for a parameter $\theta$ is a set of plausible true values of $\theta$, given the observed data. In your example, $\theta$ would be the unknown population mean.
In frequentist statistics probability represents the long-run proportion of events as the number of samples tends to infinity. In this case, the proportion of intervals that contain the unknown fixed true mean. Our confidence in any one interval covering the truth comes from this long-run performance, but each individual interval either covers or it does not. In terms of a willingness to bet you would be willing to bet $\$0.95$ and expect $\$1$ in return if it can be determined that an observed interval in question covers the truth. This is based on the long-run performance of the interval over many experiments.
It may also be useful to think in terms of a free throw percentage. Suppose you did not witness the last free throw of your favorite player. He of course either made it or missed it. Your confidence in his performance on the last free throw would be based on his free throw percentage.
It comes down to how one defines probability and what it means for something to be random. Consider a coin toss. What does it mean for a coin to be fair, to have a probability of heads equal to $50\%$? Is it fair because I have $50\%$ belief it will land on heads? Is it fair because half $(50\%)$ of the coin is heads and the other half $(50\%)$ is tails? Is it fair because in many repeated flips it lands on heads $50\%$ of the time? The last definition is the only one that is defensible.
To the frequentist it is the act of sampling that constitutes randomness - flipping a coin, shooting a basketball, etc. Probability is the long-run proportion of events - the proportion of heads, proportion of baskets, etc. If probability was applicable to a single event, what would it mean? If I flip a fair coin, catch it, and cover it on the back of my hand, what would it mean to say the coin on my hand is heads with $50\%$ probability? Is the coin in a state of superposition just because we can not see it (think Schrodinger's cat)? No. It is either heads or it is tails. All we can discuss is the performance of the coin over many flips and place our bets accordingly. Returning to the basketball example one might say, "The probability that the player makes his next shot is $75\%$." This really means, "The player makes $75\%$ of his shots and this is one such shot."
Bayesians interpret probability as the belief of the experimenter. They use this to make belief statements about the parameter itself, rather than long-run probabilities about the experiment. The problem here is that probability becomes unfalsifiable. Returning to the coin example, if we interpret probability as belief then the coin can have any probability of landing heads depending on who you ask. No one is wrong because no one can be told they are not believing the right way. We can apply probability statements to anything unknown, perhaps even the flipped coin that has already landed on my hand. None of this is defensible.
Probability is a proportion, so ask yourself, proportion of what?
The flip side of a confidence interval is the p-value. This tells us how often we would observe a result like that witnessed if a particular hypothesis is true. This sampling probability tells us how plausible the hypothesis is based on the observed data. Returning to the coin example, we will only get to flip it a finite number of times. We can test various hypotheses about the long-run performance of the coin based on our sample using p-values. Those hypotheses that are not significant at the two-sided $\alpha$ level form the $100(1-\alpha)\%$ confidence interval. It is a set in the parameter space determined by the data. This last point is often overlooked when working with a closed-form interval like a Wald interval because it is written in terms of the sample mean and standard error. I find confidence curves to be a great way to visualize frequentist inference.
